2024-06-02 20:13:53,460 INFO    StreamThr :81 [internal.py:wandb_internal():86] W&B internal server running at pid: 81, started at: 2024-06-02 20:13:53.459265
2024-06-02 20:13:53,461 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status
2024-06-02 20:13:53,843 INFO    WriterThread:81 [datastore.py:open_for_write():87] open: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/run-3mh4qp2s.wandb
2024-06-02 20:13:53,844 DEBUG   SenderThread:81 [sender.py:send():379] send: header
2024-06-02 20:13:53,847 DEBUG   SenderThread:81 [sender.py:send():379] send: run
2024-06-02 20:13:53,953 INFO    SenderThread:81 [dir_watcher.py:__init__():211] watching files in: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files
2024-06-02 20:13:53,954 INFO    SenderThread:81 [sender.py:_start_run_threads():1124] run started: 3mh4qp2s with start time 1717359233.460694
2024-06-02 20:13:53,962 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: check_version
2024-06-02 20:13:53,962 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: check_version
2024-06-02 20:13:54,052 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: run_start
2024-06-02 20:13:54,063 DEBUG   HandlerThread:81 [system_info.py:__init__():26] System info init
2024-06-02 20:13:54,063 DEBUG   HandlerThread:81 [system_info.py:__init__():41] System info init done
2024-06-02 20:13:54,063 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:13:54,064 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:13:54,064 INFO    HandlerThread:81 [system_monitor.py:probe():214] Collecting system info
2024-06-02 20:13:54,064 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:13:54,065 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:13:54,067 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:13:54,068 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:13:54,069 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:13:54,078 DEBUG   HandlerThread:81 [system_info.py:probe():150] Probing system
2024-06-02 20:13:54,080 DEBUG   HandlerThread:81 [gitlib.py:_init_repo():56] git repository is invalid
2024-06-02 20:13:54,080 DEBUG   HandlerThread:81 [system_info.py:probe():198] Probing system done
2024-06-02 20:13:54,080 DEBUG   HandlerThread:81 [system_monitor.py:probe():223] {'os': 'Linux-5.15.133+-x86_64-with-glibc2.31', 'python': '3.10.13', 'heartbeatAt': '2024-06-02T20:13:54.078341', 'startedAt': '2024-06-02T20:13:53.453169', 'docker': None, 'cuda': None, 'args': (), 'state': 'running', 'program': 'kaggle.ipynb', 'codePathLocal': None, 'root': '/kaggle/working', 'host': '8f29a1fd4865', 'username': 'root', 'executable': '/opt/conda/bin/python3.10', 'cpu_count': 2, 'cpu_count_logical': 4, 'cpu_freq': {'current': 2000.168, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2000.168, 'min': 0.0, 'max': 0.0}, {'current': 2000.168, 'min': 0.0, 'max': 0.0}, {'current': 2000.168, 'min': 0.0, 'max': 0.0}, {'current': 2000.168, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 8062.387607574463, 'used': 5643.874462127686}}, 'gpu': 'Tesla T4', 'gpu_count': 2, 'gpu_devices': [{'name': 'Tesla T4', 'memory_total': 16106127360}, {'name': 'Tesla T4', 'memory_total': 16106127360}], 'memory': {'total': 31.357563018798828}}
2024-06-02 20:13:54,080 INFO    HandlerThread:81 [system_monitor.py:probe():224] Finished collecting system info
2024-06-02 20:13:54,080 INFO    HandlerThread:81 [system_monitor.py:probe():227] Publishing system info
2024-06-02 20:13:54,080 DEBUG   HandlerThread:81 [system_info.py:_save_conda():207] Saving list of conda packages installed into the current environment
2024-06-02 20:13:54,956 INFO    Thread-12 :81 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/conda-environment.yaml
2024-06-02 20:14:09,095 ERROR   HandlerThread:81 [system_info.py:_save_conda():221] Error saving conda packages: Command '['conda', 'env', 'export']' timed out after 15 seconds
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/wandb/sdk/internal/system/system_info.py", line 214, in _save_conda
    subprocess.call(
  File "/opt/conda/lib/python3.10/subprocess.py", line 347, in call
    return p.wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1209, in wait
    return self._wait(timeout=timeout)
  File "/opt/conda/lib/python3.10/subprocess.py", line 1951, in _wait
    raise TimeoutExpired(self.args, timeout)
subprocess.TimeoutExpired: Command '['conda', 'env', 'export']' timed out after 15 seconds
2024-06-02 20:14:09,098 DEBUG   HandlerThread:81 [system_info.py:_save_conda():222] Saving conda packages done
2024-06-02 20:14:09,099 INFO    HandlerThread:81 [system_monitor.py:probe():229] Finished publishing system info
2024-06-02 20:14:09,107 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:09,107 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: keepalive
2024-06-02 20:14:09,107 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:09,108 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: keepalive
2024-06-02 20:14:09,108 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:09,108 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: keepalive
2024-06-02 20:14:09,108 DEBUG   SenderThread:81 [sender.py:send():379] send: files
2024-06-02 20:14:09,108 INFO    SenderThread:81 [sender.py:_save_file():1390] saving file wandb-metadata.json with policy now
2024-06-02 20:14:09,360 INFO    wandb-upload_0:81 [upload_job.py:push():131] Uploaded file /tmp/tmp1abbn3iqwandb/snb0frox-wandb-metadata.json
2024-06-02 20:14:09,959 INFO    Thread-12 :81 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/wandb-metadata.json
2024-06-02 20:14:10,745 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: python_packages
2024-06-02 20:14:10,746 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: python_packages
2024-06-02 20:14:10,748 DEBUG   SenderThread:81 [sender.py:send():379] send: telemetry
2024-06-02 20:14:10,825 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:14:10,825 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:14:10,842 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:14:10,864 DEBUG   SenderThread:81 [sender.py:send():379] send: config
2024-06-02 20:14:10,870 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:14:10,870 DEBUG   SenderThread:81 [sender.py:send():379] send: telemetry
2024-06-02 20:14:10,870 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:14:10,870 WARNING SenderThread:81 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-06-02 20:14:10,870 DEBUG   SenderThread:81 [sender.py:send():379] send: telemetry
2024-06-02 20:14:10,959 INFO    Thread-12 :81 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/requirements.txt
2024-06-02 20:14:10,959 INFO    Thread-12 :81 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/output.log
2024-06-02 20:14:11,862 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:14:11,862 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:14:11,862 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:14:11,863 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:14:11,863 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:14:11,863 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:14:11,864 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:14:11,864 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:14:11,872 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:14:11,872 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:14:11,872 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:14:11,873 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:14:12,960 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/output.log
2024-06-02 20:14:14,874 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:19,875 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:24,884 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:25,751 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:14:25,751 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:14:25,803 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:14:25,965 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/config.yaml
2024-06-02 20:14:30,888 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:35,889 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:40,746 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:14:40,747 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:14:40,786 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:14:41,850 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:46,852 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:51,853 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:14:55,746 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:14:55,747 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:14:55,787 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:14:57,806 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:02,808 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:07,809 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:10,746 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:15:10,747 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:15:10,787 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:15:12,823 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:17,824 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:22,825 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:25,747 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:15:25,747 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:15:25,787 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:15:28,793 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:33,794 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:38,795 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:40,747 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:15:40,747 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:15:40,787 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:15:43,876 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:48,877 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:53,878 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:15:55,747 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:15:55,747 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:15:55,787 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:15:59,826 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:04,827 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:09,828 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:10,747 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:16:10,748 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:16:10,788 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:16:14,889 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:19,890 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:24,891 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:25,747 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:16:25,748 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:16:25,788 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:16:30,793 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:35,794 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:40,747 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:16:40,748 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:16:40,788 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:16:40,813 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:45,814 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:50,815 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:16:55,748 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:16:55,748 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:16:55,788 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:16:55,866 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:00,867 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:05,868 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:10,748 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:17:10,748 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:17:10,788 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:17:11,831 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:16,832 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:21,833 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:25,748 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:17:25,748 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:17:25,788 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:17:26,840 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:31,841 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:36,843 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:40,748 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:17:40,749 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:17:40,789 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:17:42,808 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:47,809 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:52,810 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:17:55,748 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:17:55,749 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:17:55,789 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:17:58,804 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:03,805 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:08,806 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:10,748 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:18:10,749 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:18:10,789 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:18:13,850 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:18,851 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:23,852 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:25,749 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:18:25,749 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:18:25,789 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:18:28,865 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:33,866 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:38,867 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:40,749 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:18:40,749 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:18:40,789 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:18:44,340 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:18:44,340 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:18:44,340 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:18:44,341 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:18:44,341 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:18:44,342 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:18:44,343 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:18:44,343 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:18:44,345 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:18:44,381 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:18:44,381 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:18:44,381 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:18:44,381 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:18:44,381 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:18:44,382 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:18:44,382 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:18:44,383 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:18:44,390 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:18:44,390 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:18:44,390 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:18:44,390 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:18:44,391 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:49,392 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:54,393 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:18:55,749 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:18:55,750 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:18:55,750 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:18:59,871 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:04,070 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:19:04,070 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:19:04,070 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:19:04,070 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:19:04,071 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:19:04,072 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:19:04,073 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:19:04,073 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:19:04,074 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:19:04,110 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:19:04,110 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:19:04,110 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:19:04,111 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:19:04,111 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:19:04,116 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:19:04,116 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:19:04,116 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:19:04,122 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:19:04,122 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:19:04,122 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:19:04,122 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:19:04,122 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:19:04,122 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:19:04,123 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:19:04,123 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:19:04,123 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:19:04,124 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:19:04,125 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:19:04,125 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:19:04,126 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:19:04,234 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:19:04,234 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:19:04,234 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:19:04,234 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:19:04,235 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:19:04,235 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:19:04,235 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:19:04,235 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:19:04,244 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:19:04,244 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:19:04,244 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:19:04,244 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:19:04,245 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:19:04,245 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:19:04,245 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:19:04,245 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:19:04,246 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:19:04,246 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:19:04,247 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:19:04,247 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:19:04,248 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:19:04,883 DEBUG   SenderThread:81 [sender.py:send():379] send: config
2024-06-02 20:19:04,901 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:05,950 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:19:05,950 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:19:05,951 WARNING SenderThread:81 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-06-02 20:19:06,156 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/config.yaml
2024-06-02 20:19:06,157 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/output.log
2024-06-02 20:19:08,157 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/output.log
2024-06-02 20:19:10,368 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:19:10,369 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:19:10,369 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:19:10,369 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:19:10,369 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:19:10,369 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:19:10,374 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:19:10,374 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:19:10,380 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:19:10,380 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:19:10,380 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:19:10,385 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:19:10,749 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:19:10,749 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:19:10,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:19:11,872 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:16,874 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:21,875 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:25,749 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:19:25,750 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:19:25,790 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:19:27,827 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:32,828 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:37,837 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:39,200 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/config.yaml
2024-06-02 20:19:40,749 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:19:40,750 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:19:40,790 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:19:43,827 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:48,829 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:53,830 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:19:54,035 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:19:54,035 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:19:54,035 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:19:54,035 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:19:54,036 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:19:54,036 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:19:54,037 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:19:54,037 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:19:54,038 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:19:54,075 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:19:54,075 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:19:54,075 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:19:54,075 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:19:54,075 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:19:54,076 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:19:54,077 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:19:54,077 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:19:54,086 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:19:54,086 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:19:54,086 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:19:54,086 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:19:55,750 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:19:55,751 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:19:55,751 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:19:58,862 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:00,656 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:00,656 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:00,656 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:00,656 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:00,657 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:00,657 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:20:00,658 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:20:00,659 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:20:00,660 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:20:00,696 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:20:00,696 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:20:00,696 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:20:00,697 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:20:00,697 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:20:00,697 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:20:00,697 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:20:00,697 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:20:00,708 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:20:00,708 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:20:00,708 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:20:00,708 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:00,708 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:00,708 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:00,708 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:00,709 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:00,709 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:20:00,709 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:20:00,710 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:20:00,711 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:20:00,712 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:20:02,240 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:20:02,240 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:20:02,240 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:20:02,240 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:20:02,241 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:20:02,241 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:20:02,241 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:20:02,241 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:20:02,253 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:20:02,254 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:20:02,254 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:20:02,254 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:02,255 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:02,255 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:20:02,255 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:02,256 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:02,256 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:02,257 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:20:02,257 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:20:02,257 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:20:02,258 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:20:02,380 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:20:02,380 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:20:02,380 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:20:02,380 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:20:02,380 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:20:02,380 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:20:02,381 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:20:02,381 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:20:02,392 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:20:02,393 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:20:02,393 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:20:02,393 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:02,393 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:02,393 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:02,394 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:02,394 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:20:02,394 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:20:02,394 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:20:02,395 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:02,395 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:20:02,396 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:20:02,396 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:20:02,396 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:20:02,396 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:20:02,398 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:20:02,398 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:20:02,398 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:02,398 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:02,398 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:02,398 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:02,399 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:02,399 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:20:02,400 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:20:02,400 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:20:02,400 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:20:02,402 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:20:02,402 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:20:02,403 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:20:02,403 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:20:02,403 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:20:02,403 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:20:02,404 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:20:02,406 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:20:02,425 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:20:02,426 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:20:02,426 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:02,426 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:02,426 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:02,426 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:02,427 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:20:02,427 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:20:02,427 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:20:02,427 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:20:02,427 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:02,428 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:20:02,428 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:20:02,428 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:20:02,429 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:20:02,429 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:02,429 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:02,429 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:02,429 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:02,430 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:02,430 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:20:02,431 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:20:02,432 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:20:02,433 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:20:02,433 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:20:02,456 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:20:02,456 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:20:02,456 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:20:02,456 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:20:02,457 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:20:02,457 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:20:02,457 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:20:02,457 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:20:02,470 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:20:02,470 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:20:02,470 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:20:02,470 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:02,470 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:02,470 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:02,471 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:20:02,471 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:02,472 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:02,472 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:20:02,473 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:20:02,473 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:20:02,474 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:20:02,538 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: pause
2024-06-02 20:20:02,538 INFO    HandlerThread:81 [handler.py:handle_request_pause():708] stopping system metrics thread
2024-06-02 20:20:02,538 INFO    HandlerThread:81 [system_monitor.py:finish():203] Stopping system monitor
2024-06-02 20:20:02,538 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:20:02,538 DEBUG   SystemMonitor:81 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-06-02 20:20:02,538 DEBUG   SystemMonitor:81 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-06-02 20:20:02,539 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined cpu monitor
2024-06-02 20:20:02,539 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined disk monitor
2024-06-02 20:20:02,551 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined gpu monitor
2024-06-02 20:20:02,552 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined memory monitor
2024-06-02 20:20:02,552 INFO    HandlerThread:81 [interfaces.py:finish():202] Joined network monitor
2024-06-02 20:20:02,552 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: resume
2024-06-02 20:20:02,552 INFO    HandlerThread:81 [handler.py:handle_request_resume():699] starting system metrics thread
2024-06-02 20:20:02,552 INFO    HandlerThread:81 [system_monitor.py:start():194] Starting system monitor
2024-06-02 20:20:02,552 INFO    SystemMonitor:81 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-06-02 20:20:02,553 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:20:02,553 INFO    SystemMonitor:81 [interfaces.py:start():190] Started cpu monitoring
2024-06-02 20:20:02,553 INFO    SystemMonitor:81 [interfaces.py:start():190] Started disk monitoring
2024-06-02 20:20:02,554 INFO    SystemMonitor:81 [interfaces.py:start():190] Started gpu monitoring
2024-06-02 20:20:02,555 INFO    SystemMonitor:81 [interfaces.py:start():190] Started memory monitoring
2024-06-02 20:20:02,555 INFO    SystemMonitor:81 [interfaces.py:start():190] Started network monitoring
2024-06-02 20:20:03,232 DEBUG   SenderThread:81 [sender.py:send():379] send: config
2024-06-02 20:20:03,237 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:20:03,237 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:20:03,237 WARNING SenderThread:81 [sender.py:send_metric():1341] Seen metric with glob (shouldn't happen)
2024-06-02 20:20:04,209 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/output.log
2024-06-02 20:20:04,238 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:06,209 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/output.log
2024-06-02 20:20:09,361 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:10,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:20:10,753 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:20:10,753 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:20:11,270 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/config.yaml
2024-06-02 20:20:15,865 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:20,866 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:25,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:20:25,753 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:20:25,793 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:20:25,888 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:30,889 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:35,890 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:40,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:20:40,752 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:20:40,793 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:20:41,866 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:44,446 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: partial_history
2024-06-02 20:20:44,448 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:20:44,449 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:20:44,449 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:20:44,449 DEBUG   SenderThread:81 [sender.py:send():379] send: metric
2024-06-02 20:20:44,449 DEBUG   SenderThread:81 [sender.py:send():379] send: history
2024-06-02 20:20:44,449 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: summary_record
2024-06-02 20:20:44,449 INFO    SenderThread:81 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-06-02 20:20:45,283 INFO    Thread-12 :81 [dir_watcher.py:_on_file_created():271] file/dir created: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/wandb-summary.json
2024-06-02 20:20:47,450 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:52,451 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:20:55,753 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:20:55,754 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:20:55,755 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:20:57,873 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:02,555 DEBUG   SystemMonitor:81 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-06-02 20:21:02,557 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:21:03,558 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:08,559 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:10,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:21:10,752 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:21:10,792 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:21:13,850 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:15,355 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/config.yaml
2024-06-02 20:21:19,867 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:24,868 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:25,035 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: partial_history
2024-06-02 20:21:25,036 DEBUG   SenderThread:81 [sender.py:send():379] send: history
2024-06-02 20:21:25,037 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: summary_record
2024-06-02 20:21:25,039 INFO    SenderThread:81 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-06-02 20:21:25,359 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/wandb-summary.json
2024-06-02 20:21:25,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:21:25,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:21:25,753 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:21:29,869 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:32,558 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:21:35,559 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:40,560 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:40,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:21:40,753 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:21:40,793 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:21:45,835 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:50,836 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:21:55,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:21:55,753 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:21:55,793 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:21:56,805 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:01,806 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:02,561 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:22:05,261 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: partial_history
2024-06-02 20:22:05,262 DEBUG   SenderThread:81 [sender.py:send():379] send: history
2024-06-02 20:22:05,262 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: summary_record
2024-06-02 20:22:05,263 INFO    SenderThread:81 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-06-02 20:22:05,375 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/wandb-summary.json
2024-06-02 20:22:07,263 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:10,752 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:22:10,753 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:22:10,753 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:22:12,884 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:17,885 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:22,886 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:25,753 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:22:25,753 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:22:25,785 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:22:28,786 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:32,560 DEBUG   SenderThread:81 [sender.py:send():379] send: stats
2024-06-02 20:22:34,561 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:39,562 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:40,754 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:22:40,754 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:22:40,791 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages
2024-06-02 20:22:44,415 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: partial_history
2024-06-02 20:22:44,416 DEBUG   SenderThread:81 [sender.py:send():379] send: history
2024-06-02 20:22:44,416 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: summary_record
2024-06-02 20:22:44,417 INFO    SenderThread:81 [sender.py:_save_file():1390] saving file wandb-summary.json with policy end
2024-06-02 20:22:45,390 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/wandb-summary.json
2024-06-02 20:22:45,418 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:50,392 INFO    Thread-12 :81 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240602_201353-3mh4qp2s/files/output.log
2024-06-02 20:22:50,594 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:55,594 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: status_report
2024-06-02 20:22:55,753 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: stop_status
2024-06-02 20:22:55,753 DEBUG   SenderThread:81 [sender.py:send_request():406] send_request: stop_status
2024-06-02 20:22:55,756 DEBUG   HandlerThread:81 [handler.py:handle_request():146] handle_request: internal_messages