2024-03-27 19:04:18,566 INFO StreamThr :1893870 [internal.py:wandb_internal():86] W&B internal server running at pid: 1893870, started at: 2024-03-27 19:04:18.565290 2024-03-27 19:04:18,567 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status 2024-03-27 19:04:18,570 INFO WriterThread:1893870 [datastore.py:open_for_write():87] open: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/run-lwtbcr8s.wandb 2024-03-27 19:04:18,570 DEBUG SenderThread:1893870 [sender.py:send():382] send: header 2024-03-27 19:04:18,640 DEBUG SenderThread:1893870 [sender.py:send():382] send: run 2024-03-27 19:04:18,989 INFO SenderThread:1893870 [dir_watcher.py:__init__():211] watching files in: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files 2024-03-27 19:04:18,989 INFO SenderThread:1893870 [sender.py:_start_run_threads():1136] run started: lwtbcr8s with start time 1711562658.56459 2024-03-27 19:04:18,996 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: check_version 2024-03-27 19:04:18,996 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: check_version 2024-03-27 19:04:19,037 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: run_start 2024-03-27 19:04:19,050 DEBUG HandlerThread:1893870 [system_info.py:__init__():27] System info init 2024-03-27 19:04:19,050 DEBUG HandlerThread:1893870 [system_info.py:__init__():42] System info init done 2024-03-27 19:04:19,050 INFO HandlerThread:1893870 [system_monitor.py:start():194] Starting system monitor 2024-03-27 19:04:19,050 INFO SystemMonitor:1893870 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-03-27 19:04:19,051 INFO HandlerThread:1893870 [system_monitor.py:probe():214] Collecting system info 2024-03-27 19:04:19,051 INFO SystemMonitor:1893870 [interfaces.py:start():190] Started cpu monitoring 2024-03-27 19:04:19,052 INFO SystemMonitor:1893870 [interfaces.py:start():190] Started disk monitoring 2024-03-27 19:04:19,053 INFO SystemMonitor:1893870 [interfaces.py:start():190] Started gpu monitoring 2024-03-27 19:04:19,054 INFO SystemMonitor:1893870 [interfaces.py:start():190] Started memory monitoring 2024-03-27 19:04:19,056 INFO SystemMonitor:1893870 [interfaces.py:start():190] Started network monitoring 2024-03-27 19:04:19,090 DEBUG HandlerThread:1893870 [system_info.py:probe():151] Probing system 2024-03-27 19:04:19,096 DEBUG HandlerThread:1893870 [system_info.py:_probe_git():136] Probing git 2024-03-27 19:04:19,107 DEBUG HandlerThread:1893870 [system_info.py:_probe_git():144] Probing git done 2024-03-27 19:04:19,107 DEBUG HandlerThread:1893870 [system_info.py:probe():199] Probing system done 2024-03-27 19:04:19,108 DEBUG HandlerThread:1893870 [system_monitor.py:probe():223] {'os': 'Linux-5.4.0-166-generic-x86_64-with-glibc2.29', 'python': '3.8.10', 'heartbeatAt': '2024-03-27T18:04:19.090782', 'startedAt': '2024-03-27T18:04:18.560848', 'docker': None, 'cuda': None, 'args': ('--model_name_or_path=distil-whisper/distil-large-v3', '--dataset_name=mozilla-foundation/common_voice_16_1', '--dataset_config_name=hi', '--language=hindi', '--train_split_name=train+validation', '--eval_split_name=test', '--max_steps=5000', '--output_dir=./', '--per_device_train_batch_size=32', '--per_device_eval_batch_size=32', '--logging_steps=25', '--learning_rate=1e-4', '--warmup_steps=500', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--save_total_limit=1', '--generation_max_length=225', '--preprocessing_num_workers=1', '--dataloader_num_workers=4', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': 'run_speech_recognition_seq2seq.py', 'codePathLocal': 'run_speech_recognition_seq2seq.py', 'codePath': 'run_speech_recognition_seq2seq.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/distil-large-v3-hi-ft', 'commit': '40c686df113c0e98e7363c1bd523f58d11848fc0'}, 'email': 'sanchit@huggingface.co', 'root': '/home/sanchit/distil-large-v3-hi-ft', 'host': 'hf-dgx-01', 'username': 'sanchit', 'executable': '/home/sanchit/hf/bin/python', 'cpu_count': 64, 'cpu_count_logical': 128, 'cpu_freq': {'current': 2290.4996484375006, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 2186.729, 'min': 1500.0, 'max': 2250.0}, {'current': 1795.542, 'min': 1500.0, 'max': 2250.0}, {'current': 1792.045, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.339, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.496, 'min': 1500.0, 'max': 2250.0}, {'current': 1742.02, 'min': 1500.0, 'max': 2250.0}, {'current': 1692.095, 'min': 1500.0, 'max': 2250.0}, {'current': 1658.911, 'min': 1500.0, 'max': 2250.0}, {'current': 1606.212, 'min': 1500.0, 'max': 2250.0}, {'current': 1606.808, 'min': 1500.0, 'max': 2250.0}, {'current': 1606.448, 'min': 1500.0, 'max': 2250.0}, {'current': 2094.446, 'min': 1500.0, 'max': 2250.0}, {'current': 1734.944, 'min': 1500.0, 'max': 2250.0}, {'current': 1736.215, 'min': 1500.0, 'max': 2250.0}, {'current': 2489.415, 'min': 1500.0, 'max': 2250.0}, {'current': 1737.217, 'min': 1500.0, 'max': 2250.0}, {'current': 1695.232, 'min': 1500.0, 'max': 2250.0}, {'current': 2416.542, 'min': 1500.0, 'max': 2250.0}, {'current': 1737.614, 'min': 1500.0, 'max': 2250.0}, {'current': 1599.452, 'min': 1500.0, 'max': 2250.0}, {'current': 1772.758, 'min': 1500.0, 'max': 2250.0}, {'current': 1738.087, 'min': 1500.0, 'max': 2250.0}, {'current': 2311.378, 'min': 1500.0, 'max': 2250.0}, {'current': 1735.599, 'min': 1500.0, 'max': 2250.0}, {'current': 1620.967, 'min': 1500.0, 'max': 2250.0}, {'current': 1864.863, 'min': 1500.0, 'max': 2250.0}, {'current': 1729.177, 'min': 1500.0, 'max': 2250.0}, {'current': 1728.804, 'min': 1500.0, 'max': 2250.0}, {'current': 1788.133, 'min': 1500.0, 'max': 2250.0}, {'current': 1795.255, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.306, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.529, 'min': 1500.0, 'max': 2250.0}, {'current': 1733.742, 'min': 1500.0, 'max': 2250.0}, {'current': 2473.813, 'min': 1500.0, 'max': 2250.0}, {'current': 1932.309, 'min': 1500.0, 'max': 2250.0}, {'current': 1727.805, 'min': 1500.0, 'max': 2250.0}, {'current': 1730.935, 'min': 1500.0, 'max': 2250.0}, {'current': 1729.65, 'min': 1500.0, 'max': 2250.0}, {'current': 1730.989, 'min': 1500.0, 'max': 2250.0}, {'current': 2002.554, 'min': 1500.0, 'max': 2250.0}, {'current': 2089.927, 'min': 1500.0, 'max': 2250.0}, {'current': 1727.843, 'min': 1500.0, 'max': 2250.0}, {'current': 1728.434, 'min': 1500.0, 'max': 2250.0}, {'current': 2271.01, 'min': 1500.0, 'max': 2250.0}, {'current': 3370.134, 'min': 1500.0, 'max': 2250.0}, {'current': 1687.104, 'min': 1500.0, 'max': 2250.0}, {'current': 1688.345, 'min': 1500.0, 'max': 2250.0}, {'current': 1694.029, 'min': 1500.0, 'max': 2250.0}, {'current': 3295.66, 'min': 1500.0, 'max': 2250.0}, {'current': 1684.621, 'min': 1500.0, 'max': 2250.0}, {'current': 1686.668, 'min': 1500.0, 'max': 2250.0}, {'current': 1687.585, 'min': 1500.0, 'max': 2250.0}, {'current': 1727.907, 'min': 1500.0, 'max': 2250.0}, {'current': 1728.023, 'min': 1500.0, 'max': 2250.0}, {'current': 1727.835, 'min': 1500.0, 'max': 2250.0}, {'current': 2090.043, 'min': 1500.0, 'max': 2250.0}, {'current': 2245.903, 'min': 1500.0, 'max': 2250.0}, {'current': 2202.881, 'min': 1500.0, 'max': 2250.0}, {'current': 2284.838, 'min': 1500.0, 'max': 2250.0}, {'current': 1598.829, 'min': 1500.0, 'max': 2250.0}, {'current': 1656.082, 'min': 1500.0, 'max': 2250.0}, {'current': 1661.638, 'min': 1500.0, 'max': 2250.0}, {'current': 1649.714, 'min': 1500.0, 'max': 2250.0}, {'current': 1655.626, 'min': 1500.0, 'max': 2250.0}, {'current': 3027.152, 'min': 1500.0, 'max': 2250.0}, {'current': 2017.291, 'min': 1500.0, 'max': 2250.0}, {'current': 2012.466, 'min': 1500.0, 'max': 2250.0}, {'current': 1990.236, 'min': 1500.0, 'max': 2250.0}, {'current': 2114.394, 'min': 1500.0, 'max': 2250.0}, {'current': 2095.433, 'min': 1500.0, 'max': 2250.0}, {'current': 2096.911, 'min': 1500.0, 'max': 2250.0}, {'current': 1840.403, 'min': 1500.0, 'max': 2250.0}, {'current': 2203.59, 'min': 1500.0, 'max': 2250.0}, {'current': 2185.592, 'min': 1500.0, 'max': 2250.0}, {'current': 2201.94, 'min': 1500.0, 'max': 2250.0}, {'current': 2018.775, 'min': 1500.0, 'max': 2250.0}, {'current': 2200.874, 'min': 1500.0, 'max': 2250.0}, {'current': 1873.351, 'min': 1500.0, 'max': 2250.0}, {'current': 3369.243, 'min': 1500.0, 'max': 2250.0}, {'current': 2235.133, 'min': 1500.0, 'max': 2250.0}, {'current': 3367.352, 'min': 1500.0, 'max': 2250.0}, {'current': 3364.254, 'min': 1500.0, 'max': 2250.0}, {'current': 2202.453, 'min': 1500.0, 'max': 2250.0}, {'current': 2038.768, 'min': 1500.0, 'max': 2250.0}, {'current': 2233.914, 'min': 1500.0, 'max': 2250.0}, {'current': 2182.285, 'min': 1500.0, 'max': 2250.0}, {'current': 2302.527, 'min': 1500.0, 'max': 2250.0}, {'current': 2185.247, 'min': 1500.0, 'max': 2250.0}, {'current': 2132.843, 'min': 1500.0, 'max': 2250.0}, {'current': 2971.398, 'min': 1500.0, 'max': 2250.0}, {'current': 2055.752, 'min': 1500.0, 'max': 2250.0}, {'current': 1567.353, 'min': 1500.0, 'max': 2250.0}, {'current': 1609.825, 'min': 1500.0, 'max': 2250.0}, {'current': 1606.242, 'min': 1500.0, 'max': 2250.0}, {'current': 1610.589, 'min': 1500.0, 'max': 2250.0}, {'current': 1617.499, 'min': 1500.0, 'max': 2250.0}, {'current': 2398.668, 'min': 1500.0, 'max': 2250.0}, {'current': 3020.272, 'min': 1500.0, 'max': 2250.0}, {'current': 2053.074, 'min': 1500.0, 'max': 2250.0}, {'current': 2141.083, 'min': 1500.0, 'max': 2250.0}, {'current': 2182.762, 'min': 1500.0, 'max': 2250.0}, {'current': 2190.047, 'min': 1500.0, 'max': 2250.0}, {'current': 2209.589, 'min': 1500.0, 'max': 2250.0}, {'current': 2072.985, 'min': 1500.0, 'max': 2250.0}, {'current': 2427.726, 'min': 1500.0, 'max': 2250.0}, {'current': 2205.137, 'min': 1500.0, 'max': 2250.0}, {'current': 2197.942, 'min': 1500.0, 'max': 2250.0}, {'current': 2208.468, 'min': 1500.0, 'max': 2250.0}, {'current': 3307.897, 'min': 1500.0, 'max': 2250.0}, {'current': 2210.539, 'min': 1500.0, 'max': 2250.0}, {'current': 2580.208, 'min': 1500.0, 'max': 2250.0}, {'current': 2487.416, 'min': 1500.0, 'max': 2250.0}, {'current': 3360.967, 'min': 1500.0, 'max': 2250.0}, {'current': 2409.573, 'min': 1500.0, 'max': 2250.0}, {'current': 2399.184, 'min': 1500.0, 'max': 2250.0}, {'current': 2400.551, 'min': 1500.0, 'max': 2250.0}, {'current': 2453.642, 'min': 1500.0, 'max': 2250.0}, {'current': 1814.426, 'min': 1500.0, 'max': 2250.0}, {'current': 2440.557, 'min': 1500.0, 'max': 2250.0}, {'current': 2422.632, 'min': 1500.0, 'max': 2250.0}, {'current': 1919.862, 'min': 1500.0, 'max': 2250.0}, {'current': 2519.195, 'min': 1500.0, 'max': 2250.0}, {'current': 1666.938, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.157, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.328, 'min': 1500.0, 'max': 2250.0}, {'current': 1797.233, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.218, 'min': 1500.0, 'max': 2250.0}, {'current': 1790.531, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 1757.8785285949707, 'used': 1614.0720481872559}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 5, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA DGX Display', 'memory_total': 4294967296}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 503.5396919250488}} 2024-03-27 19:04:19,108 INFO HandlerThread:1893870 [system_monitor.py:probe():224] Finished collecting system info 2024-03-27 19:04:19,108 INFO HandlerThread:1893870 [system_monitor.py:probe():227] Publishing system info 2024-03-27 19:04:19,109 INFO HandlerThread:1893870 [system_monitor.py:probe():229] Finished publishing system info 2024-03-27 19:04:19,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:19,113 DEBUG SenderThread:1893870 [sender.py:send():382] send: files 2024-03-27 19:04:19,113 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now 2024-03-27 19:04:19,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: python_packages 2024-03-27 19:04:19,118 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: python_packages 2024-03-27 19:04:19,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:04:19,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:04:19,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:04:19,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:19,280 DEBUG SenderThread:1893870 [sender.py:send():382] send: telemetry 2024-03-27 19:04:19,281 DEBUG SenderThread:1893870 [sender.py:send():382] send: config 2024-03-27 19:04:19,281 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 19:04:19,281 DEBUG SenderThread:1893870 [sender.py:send():382] send: telemetry 2024-03-27 19:04:19,281 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 19:04:19,281 WARNING SenderThread:1893870 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-03-27 19:04:19,281 DEBUG SenderThread:1893870 [sender.py:send():382] send: telemetry 2024-03-27 19:04:19,625 INFO wandb-upload_0:1893870 [upload_job.py:push():131] Uploaded file /tmp/tmp4zjika_rwandb/yd0vpcfo-wandb-metadata.json 2024-03-27 19:04:19,991 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-metadata.json 2024-03-27 19:04:19,991 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/requirements.txt 2024-03-27 19:04:19,991 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:21,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:21,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:21,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:23,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:23,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:24,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:04:26,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:26,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:28,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:28,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:29,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:04:31,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:31,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:33,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:33,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:33,993 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:34,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:04:34,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:04:34,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:04:34,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:04:35,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:35,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:37,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:37,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:37,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:39,995 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:40,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:04:40,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:40,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:42,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:42,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:43,996 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:45,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:04:45,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:45,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:45,997 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:47,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:47,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:49,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:04:49,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:04:49,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:04:49,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:50,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:50,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:50,804 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:04:50,999 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/config.yaml 2024-03-27 19:04:51,999 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:52,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:52,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:56,001 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:56,107 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:04:56,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:56,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:56,561 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:04:58,002 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:04:58,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:04:58,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:00,002 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:00,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:00,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:02,026 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:02,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:02,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:04,003 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:04,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:05:04,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:05:04,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:05:05,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:05,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:06,004 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:07,567 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:07,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:07,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:09,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:09,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:10,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:11,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:11,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:12,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:13,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:14,006 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:14,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:14,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:16,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:16,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:18,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:18,608 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:19,057 DEBUG SystemMonitor:1893870 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-03-27 19:05:19,059 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:05:19,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:05:19,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:05:19,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:05:19,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:19,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:20,008 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:21,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:21,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:23,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:24,009 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:24,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:24,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:26,009 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:26,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:26,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:28,010 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:29,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:29,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:29,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:33,689 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:05:33,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:33,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:34,011 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:34,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:05:34,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:05:34,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:05:34,300 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:36,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:36,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:38,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:38,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:40,013 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:40,219 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:41,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:41,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:43,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:43,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:44,013 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:46,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:46,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:46,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:48,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:48,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:49,061 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:05:49,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:05:49,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:05:49,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:05:50,015 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:50,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:50,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:51,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:53,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:53,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:55,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:55,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:56,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:05:57,202 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:05:58,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:05:58,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:00,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:00,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:01,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:06:01,277 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 19:06:01,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 19:06:01,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 19:06:01,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 19:06:01,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:06:01,278 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:06:01,279 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:06:02,017 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:02,018 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:06:02,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:03,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:03,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:04,018 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:04,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:06:04,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:06:04,120 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:06:05,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:05,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:08,019 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:08,224 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:08,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:08,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:12,549 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:06:12,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:12,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:13,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:14,021 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:15,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:15,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:17,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:17,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:18,294 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:19,063 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:06:19,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:06:19,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:06:19,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:06:20,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:20,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:20,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:22,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:22,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:23,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:25,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:25,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:26,023 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:27,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:27,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:28,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:29,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/config.yaml 2024-03-27 19:06:30,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:30,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:32,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:32,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:32,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:34,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:06:34,120 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:06:34,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:06:34,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:35,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:35,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:37,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:37,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:38,026 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:39,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:40,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:40,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:42,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:42,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:44,028 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:44,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:45,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:45,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:47,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:47,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:49,064 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:06:49,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:06:49,119 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:06:49,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:06:49,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:51,181 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:06:51,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:51,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:52,029 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:54,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:54,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:54,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:06:56,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:56,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:58,031 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:06:59,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:59,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:06:59,707 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:01,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:01,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:04,032 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:04,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:07:04,120 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:07:04,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:07:04,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:04,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:05,344 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:06,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:06,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:09,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:09,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:10,034 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:10,797 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:11,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:11,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:14,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:14,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:15,851 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:16,035 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:16,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:16,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:19,066 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:07:19,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:07:19,120 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:07:19,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:07:19,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:19,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:20,900 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:21,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:21,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:22,037 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:24,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:24,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:25,901 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:26,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:26,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:28,038 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:30,090 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:07:30,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:30,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:30,958 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:32,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:32,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:34,040 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:34,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:07:34,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:07:34,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:07:34,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:34,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:36,344 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:37,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:37,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:39,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:39,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:40,042 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:41,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:42,051 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:42,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:44,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:44,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:46,043 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:47,105 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:48,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:48,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:49,068 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:07:49,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:07:49,120 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:07:49,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:07:50,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:50,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:52,045 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:52,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:53,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:53,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:55,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:55,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:57,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:07:58,047 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:07:58,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:07:58,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:00,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:00,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:02,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:03,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:03,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:04,049 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:04,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:08:04,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:08:04,121 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:08:05,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:05,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:07,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:08,731 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:08:08,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:08,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:10,051 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:11,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:11,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:13,318 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:13,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:13,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:16,052 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:16,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:16,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:18,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:18,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:18,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:19,069 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:08:19,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:08:19,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:08:19,121 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:08:21,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:21,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:22,054 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:23,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:23,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:23,687 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:26,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:26,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:28,056 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:28,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:28,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:28,739 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:31,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:31,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:32,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:08:32,870 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:08:32,870 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:08:32,871 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:08:33,058 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:08:33,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:33,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:33,871 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:34,058 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:34,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:08:34,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:08:34,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:08:36,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:36,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:38,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:38,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:39,862 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:40,060 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:41,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:41,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:43,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:43,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:44,863 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:46,062 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:46,988 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:08:47,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:47,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:49,071 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:08:49,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:08:49,121 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:08:49,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:08:50,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:50,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:50,287 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:52,064 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:52,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:52,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:54,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:54,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:55,968 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:08:56,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:56,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:58,066 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:08:58,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:08:58,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:01,030 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:01,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:01,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:03,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:03,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:04,068 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:04,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:09:04,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:09:04,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:09:06,347 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:06,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:06,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:08,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:08,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:10,069 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:11,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:11,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:12,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:13,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:13,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:15,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:15,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:16,071 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:17,205 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:18,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:18,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:19,073 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:09:19,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:09:19,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:09:19,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:09:20,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:20,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:22,075 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:22,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:24,380 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:09:24,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:24,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:27,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:27,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:27,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:28,077 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:29,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:29,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:32,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:32,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:32,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:34,079 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:34,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:09:34,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:09:34,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:09:34,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:34,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:37,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:37,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:38,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:39,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:39,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:40,081 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:42,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:42,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:43,588 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:44,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:44,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:46,083 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:47,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:47,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:48,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:49,075 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:09:49,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:09:49,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:09:49,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:09:49,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:49,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:52,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:52,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:53,710 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:54,085 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:09:54,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:54,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:56,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:56,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:58,771 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:09:59,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:09:59,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:00,087 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:03,234 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:10:03,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:03,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:03,772 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:04,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:10:04,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:10:04,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:10:06,089 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:06,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:06,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:08,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:08,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:09,314 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:10,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:10,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:12,091 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:13,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:13,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:14,883 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:15,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:15,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:18,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:18,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:18,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:19,076 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:10:19,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:10:19,121 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:10:19,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:10:20,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:20,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:20,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:23,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:23,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:24,096 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:25,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:25,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:25,375 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:27,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:27,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:30,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:30,098 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:30,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:30,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:32,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:32,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:34,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:10:34,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:10:34,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:10:34,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:34,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:35,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:36,100 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:37,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:37,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:40,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:40,798 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:10:40,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:40,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:42,102 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:42,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:42,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:45,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:45,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:45,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:47,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:47,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:48,104 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:49,078 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:10:49,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:10:49,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:10:49,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:10:49,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:49,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:51,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:52,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:52,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:54,107 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:10:54,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:54,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:56,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:56,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:56,775 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:10:59,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:10:59,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:00,109 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:01,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:01,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:01,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:04,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:11:04,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:11:04,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:11:04,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:04,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:05,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:11:05,258 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:11:05,258 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:11:05,258 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:11:06,111 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:11:06,111 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:06,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:06,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:07,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:08,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:08,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:11,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:11,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:12,113 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:12,502 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:13,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:13,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:16,902 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:11:16,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:16,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:17,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:18,115 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:19,080 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:11:19,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:11:19,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:11:19,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:11:19,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:19,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:21,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:21,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:23,319 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:24,117 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:24,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:24,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:26,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:26,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:28,626 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:29,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:29,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:30,119 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:31,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:31,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:33,986 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:34,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:11:34,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:11:34,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:11:34,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:34,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:36,121 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:36,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:36,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:39,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:39,038 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:39,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:40,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:41,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:41,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:44,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:44,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:44,109 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:44,124 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:46,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:46,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:46,124 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:48,125 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:48,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:48,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:49,080 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:11:49,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:11:49,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:11:49,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:11:50,800 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:51,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:51,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:52,126 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:54,631 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:11:54,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:54,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:55,880 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:11:57,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:57,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:58,129 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:11:59,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:11:59,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:01,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:01,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:01,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:04,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:12:04,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:12:04,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:12:04,131 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:04,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:04,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:06,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:06,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:07,276 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:08,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:08,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:10,133 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:11,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:11,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:12,629 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:13,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:13,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:16,135 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:16,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:16,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:17,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:18,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:18,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:19,084 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:12:19,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:12:19,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:12:19,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:12:21,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:21,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:22,137 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:22,969 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:23,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:23,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:26,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:26,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:28,019 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:28,139 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:28,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:28,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:33,004 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:12:33,020 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:33,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:33,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:34,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:12:34,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:12:34,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:12:34,141 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:35,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:35,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:37,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:37,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:38,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:40,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:40,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:40,144 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:42,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:42,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:44,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:44,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:44,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:46,146 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:46,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:46,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:49,084 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:12:49,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:12:49,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:12:49,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:12:49,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:49,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:50,941 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:51,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:51,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:52,148 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:55,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:55,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:56,775 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:12:57,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:57,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:58,150 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:12:59,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:12:59,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:01,834 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:02,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:02,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:04,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:13:04,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:13:04,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:13:04,152 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:05,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:05,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:07,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:07,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:07,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:10,682 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:13:10,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:10,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:12,155 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:12,961 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:13,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:13,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:15,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:15,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:17,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:17,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:18,009 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:18,157 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:19,085 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:13:19,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:13:19,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:13:19,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:13:20,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:20,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:22,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:13:22,132 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:13:22,133 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:13:22,135 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:13:22,159 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:13:22,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:22,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:23,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:24,159 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:25,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:25,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:27,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:27,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:29,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:30,161 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:30,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:30,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:32,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:32,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:34,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:13:34,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:13:34,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:13:34,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:35,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:35,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:36,163 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:37,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:37,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:39,347 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:40,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:40,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:42,166 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:42,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:42,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:44,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:45,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:45,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:48,168 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:49,087 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:13:49,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:13:49,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:13:49,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:13:49,411 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:13:49,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:49,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:50,264 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:52,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:52,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:54,170 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:13:54,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:54,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:55,582 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:13:57,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:57,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:59,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:13:59,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:00,172 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:00,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:02,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:02,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:04,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:14:04,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:14:04,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:14:04,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:04,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:05,698 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:06,174 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:07,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:07,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:09,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:09,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:10,699 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:12,175 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:12,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:12,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:14,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:14,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:15,772 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:17,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:17,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:18,178 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:19,089 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:14:19,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:14:19,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:14:19,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:14:19,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:19,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:21,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:22,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:22,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:24,180 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:24,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:24,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:26,888 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:28,076 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:14:28,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:28,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:30,182 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:30,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:30,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:31,936 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:32,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:32,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:34,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:14:34,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:14:34,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:14:35,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:35,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:36,184 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:36,988 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:37,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:37,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:40,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:40,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:42,056 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:42,186 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:42,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:42,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:45,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:45,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:47,057 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:47,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:47,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:48,188 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:49,091 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:14:49,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:14:49,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:14:49,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:14:50,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:51,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:52,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:14:53,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:53,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:54,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:14:56,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:56,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:58,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:58,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:14:58,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:00,192 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:00,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:00,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:03,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:03,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:03,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:04,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:15:04,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:15:04,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:15:06,423 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:15:06,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:06,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:07,195 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:08,298 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:09,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:09,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:11,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:11,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:13,197 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:13,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:14,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:14,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:16,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:16,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:18,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:19,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:19,092 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:15:19,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:19,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:15:19,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:15:19,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:15:19,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:21,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:21,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:23,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:24,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:24,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:25,201 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:26,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:26,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:28,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:28,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:28,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:31,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:31,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:31,203 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:33,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:33,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:34,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:34,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:15:34,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:15:34,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:15:36,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:36,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:37,206 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:38,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:39,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:39,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:41,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:41,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:43,209 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:43,676 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:44,609 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:15:44,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:44,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:47,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:47,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:48,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:49,093 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:15:49,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:15:49,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:15:49,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:15:49,212 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:49,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:49,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:52,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:52,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:53,878 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:53,950 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:15:53,951 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:15:53,952 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:15:53,952 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:15:54,214 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:15:54,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:54,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:55,214 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:56,220 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:15:57,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:57,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:58,953 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:15:59,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:15:59,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:01,222 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:02,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:02,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:04,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:16:04,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:16:04,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:16:04,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:04,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:04,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:06,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:06,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:07,225 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:09,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:09,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:10,013 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:11,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:11,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:13,228 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:14,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:14,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:15,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:16,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:16,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:19,095 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:16:19,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:16:19,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:16:19,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:16:19,230 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:19,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:19,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:20,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:23,037 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:16:23,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:23,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:26,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:26,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:26,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:27,233 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:28,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:28,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:31,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:31,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:31,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:33,235 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:33,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:33,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:34,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:16:34,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:16:34,125 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:16:36,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:36,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:36,298 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:38,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:38,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:39,243 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:41,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:41,298 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:41,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:43,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:43,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:45,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:46,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:46,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:46,367 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:48,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:48,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:49,096 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:16:49,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:16:49,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:16:49,125 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:16:51,247 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:51,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:51,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:52,364 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:53,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:53,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:56,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:56,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:57,250 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:16:57,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:16:58,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:16:58,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:01,917 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:17:01,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:01,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:02,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:03,252 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:04,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:17:04,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:17:04,125 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:17:04,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:04,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:06,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:06,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:07,621 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:09,255 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:09,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:09,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:11,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:11,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:12,622 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:14,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:14,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:15,257 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:16,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:17,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:17,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:19,098 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:17:19,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:17:19,125 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:17:19,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:17:19,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:19,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:21,259 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:21,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:21,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:23,356 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:24,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:24,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:26,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:27,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:27,261 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:28,831 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:29,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:29,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:31,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:32,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:33,265 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:33,893 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:34,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:17:34,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:17:34,125 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:17:34,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:34,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:37,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:37,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:38,958 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:39,267 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:40,552 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:17:40,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:40,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:43,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:43,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:44,009 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:45,269 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:45,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:45,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:48,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:48,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:49,100 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:17:49,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:49,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:17:49,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:17:49,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:17:49,271 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:50,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:50,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:53,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:53,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:54,331 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:17:55,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:55,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:57,273 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:17:58,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:58,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:17:59,331 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:00,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:00,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:01,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:03,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:03,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:04,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:18:04,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:18:04,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:18:04,341 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:05,276 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:05,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:05,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:07,276 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:08,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:08,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:10,341 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:10,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:10,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:11,099 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:18:11,100 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:18:11,101 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:18:11,102 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:18:11,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:18:11,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:12,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:13,279 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:13,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:13,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:15,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:15,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:16,103 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:19,101 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:18:19,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:18:19,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:18:19,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:18:19,248 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:18:19,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:19,280 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:19,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:21,280 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:22,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:22,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:23,287 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:24,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:24,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:27,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:27,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:27,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:29,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:29,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:29,288 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:32,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:32,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:32,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:34,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:34,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:34,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:18:34,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:18:34,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:18:36,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:36,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:37,291 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:37,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:38,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:38,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:41,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:41,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:42,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:43,293 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:44,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:44,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:47,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:47,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:47,508 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:49,102 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:18:49,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:18:49,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:18:49,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:18:49,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:49,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:49,295 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:52,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:52,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:52,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:54,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:54,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:55,297 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:18:57,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:18:57,754 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:18:57,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:18:57,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:00,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:00,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:01,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:02,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:02,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:02,612 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:04,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:19:04,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:19:04,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:19:05,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:05,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:07,301 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:07,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:07,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:07,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:10,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:10,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:12,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:12,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:12,731 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:13,303 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:15,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:15,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:17,800 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:17,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:17,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:19,104 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:19:19,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:19:19,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:19:19,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:19:19,305 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:20,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:20,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:22,854 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:22,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:22,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:24,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:24,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:25,306 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:27,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:27,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:27,855 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:29,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:29,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:31,308 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:31,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:31,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:32,913 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:34,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:19:34,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:19:34,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:19:35,284 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:19:35,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:35,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:37,310 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:37,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:38,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:38,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:40,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:40,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:42,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:42,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:43,034 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:43,312 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:44,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:44,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:47,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:47,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:48,087 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:49,106 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:19:49,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:19:49,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:19:49,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:19:49,314 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:50,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:50,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:52,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:52,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:53,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:19:55,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:55,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:55,315 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:19:57,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:57,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:19:58,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:00,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:00,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:01,317 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:02,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:02,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:03,220 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:04,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:20:04,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:20:04,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:20:04,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:04,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:06,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:06,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:07,323 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:08,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:08,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:08,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:12,526 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:20:12,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:12,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:13,325 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:13,340 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:15,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:15,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:17,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:17,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:18,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:19,107 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:20:19,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:20:19,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:20:19,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:20:19,327 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:20,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:20,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:22,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:22,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:23,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:25,330 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:25,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:25,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:27,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:27,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:28,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:30,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:30,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:31,332 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:32,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:32,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:33,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:34,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:20:34,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:20:34,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:20:35,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:35,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:37,334 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:37,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:37,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:38,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:40,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:40,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:41,719 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:20:41,720 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:20:41,721 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:20:41,721 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:20:42,335 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:20:42,335 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:42,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:42,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:43,336 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:43,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:45,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:45,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:47,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:47,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:49,108 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:20:49,109 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:49,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:20:49,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:20:49,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:20:49,339 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:51,501 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:20:51,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:51,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:54,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:54,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:54,766 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:20:55,341 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:20:56,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:56,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:58,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:58,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:20:59,767 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:01,342 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:01,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:01,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:03,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:03,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:04,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:21:04,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:21:04,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:21:05,317 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:06,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:06,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:07,344 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:08,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:08,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:10,888 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:11,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:11,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:13,347 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:13,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:13,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:15,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:16,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:16,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:18,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:18,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:19,110 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:21:19,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:21:19,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:21:19,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:21:19,349 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:21,334 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:21,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:21,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:23,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:23,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:25,351 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:26,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:26,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:27,060 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:30,537 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:21:30,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:30,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:31,353 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:32,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:33,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:33,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:34,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:21:34,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:21:34,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:21:35,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:35,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:37,199 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:37,355 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:38,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:38,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:40,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:40,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:42,200 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:43,357 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:43,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:43,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:45,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:45,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:47,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:48,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:48,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:49,111 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:21:49,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:21:49,126 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:21:49,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:21:49,359 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:50,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:50,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:52,326 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:53,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:53,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:55,361 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:21:55,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:55,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:57,385 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:21:57,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:57,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:21:59,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:00,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:01,363 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:02,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:03,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:03,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:04,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:22:04,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:22:04,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:22:05,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:05,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:07,366 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:07,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:08,705 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:22:08,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:08,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:11,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:11,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:12,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:13,368 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:13,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:13,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:16,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:16,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:17,597 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:18,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:18,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:19,113 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:22:19,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:22:19,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:22:19,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:22:20,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:20,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:21,371 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:23,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:24,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:24,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:26,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:26,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:27,373 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:28,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:28,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:28,740 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:31,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:31,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:33,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:33,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:33,375 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:33,811 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:34,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:22:34,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:22:34,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:22:36,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:36,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:38,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:38,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:38,865 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:39,377 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:41,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:41,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:43,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:43,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:43,955 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:45,379 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:47,324 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:22:47,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:47,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:48,955 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:49,115 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:22:49,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:22:49,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:22:49,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:22:49,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:49,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:51,381 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:51,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:51,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:54,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:22:54,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:54,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:56,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:56,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:57,383 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:22:59,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:22:59,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:00,098 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:01,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:01,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:03,385 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:04,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:23:04,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:23:04,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:23:04,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:04,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:05,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:06,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:06,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:08,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:08,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:09,387 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:10,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:10,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:11,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:12,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:12,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:13,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:23:13,408 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:23:13,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:23:13,410 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:23:14,389 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:23:15,390 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:15,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:15,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:16,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:17,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:17,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:19,116 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:23:19,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:23:19,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:23:19,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:23:20,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:20,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:21,392 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:21,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:24,408 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:23:24,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:24,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:26,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:27,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:27,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:27,394 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:29,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:29,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:31,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:31,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:31,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:33,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:33,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:33,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:34,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:23:34,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:23:34,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:23:36,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:36,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:37,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:38,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:38,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:39,399 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:41,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:41,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:42,690 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:43,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:43,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:45,401 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:46,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:46,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:47,784 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:48,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:48,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:49,118 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:23:49,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:23:49,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:23:49,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:23:51,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:51,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:51,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:53,674 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:53,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:53,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:57,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:57,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:57,405 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:23:58,953 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:23:59,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:23:59,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:02,540 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:24:02,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:02,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:03,408 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:04,032 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:04,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:24:04,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:24:04,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:24:05,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:05,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:07,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:07,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:09,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:09,410 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:10,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:10,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:12,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:12,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:14,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:15,413 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:15,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:15,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:17,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:17,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:19,119 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:24:19,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:19,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:24:19,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:24:19,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:24:20,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:20,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:22,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:22,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:23,416 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:24,797 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:25,417 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:25,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:25,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:27,417 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:27,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:27,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:29,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:29,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:30,300 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:31,419 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:32,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:32,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:33,420 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:34,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:24:34,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:24:34,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:24:34,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:34,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:36,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:36,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:36,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:37,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:39,941 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:24:39,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:39,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:41,764 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:42,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:42,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:43,424 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:44,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:44,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:47,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:47,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:47,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:49,121 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:24:49,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:24:49,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:24:49,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:24:49,426 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:49,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:49,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:52,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:52,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:52,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:54,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:54,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:55,429 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:24:57,604 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:24:57,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:57,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:59,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:24:59,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:01,430 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:02,671 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:03,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:03,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:04,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:25:04,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:25:04,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:25:05,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:05,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:07,433 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:07,743 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:08,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:08,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:10,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:10,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:13,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:13,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:13,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:13,435 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:15,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:15,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:18,067 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:18,626 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:25:18,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:18,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:19,122 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:25:19,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:25:19,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:25:19,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:25:21,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:21,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:23,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:23,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:23,330 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:25,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:25,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:27,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:27,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:29,079 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:29,441 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:30,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:30,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:32,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:33,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:33,442 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:34,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:25:34,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:25:34,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:25:34,315 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:35,104 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:25:35,106 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:25:35,106 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:25:35,106 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:25:35,443 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:25:36,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:36,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:37,444 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:38,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:38,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:40,107 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:40,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:40,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:42,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:42,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:43,446 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:45,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:45,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:46,104 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:47,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:47,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:49,125 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:25:49,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:25:49,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:25:49,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:25:49,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:49,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:49,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:51,296 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:52,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:52,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:55,450 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:25:56,348 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:25:56,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:56,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:57,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:25:58,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:25:58,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:01,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:01,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:01,453 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:02,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:03,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:03,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:04,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:26:04,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:26:04,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:26:06,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:06,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:07,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:07,455 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:08,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:08,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:11,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:11,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:12,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:13,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:13,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:13,459 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:16,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:16,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:17,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:18,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:18,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:19,125 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:26:19,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:26:19,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:26:19,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:26:19,466 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:20,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:20,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:23,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:23,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:23,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:25,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:25,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:25,469 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:27,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:27,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:28,898 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:29,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:30,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:31,471 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:33,730 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:26:33,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:33,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:33,975 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:34,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:26:34,128 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:26:34,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:26:35,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:35,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:37,474 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:38,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:38,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:39,030 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:40,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:40,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:43,476 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:43,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:43,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:44,097 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:45,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:45,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:48,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:48,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:49,128 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:26:49,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:49,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:26:49,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:26:49,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:26:49,478 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:50,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:50,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:53,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:53,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:54,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:26:55,480 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:26:55,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:55,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:57,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:57,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:26:59,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:00,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:00,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:01,482 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:02,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:02,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:04,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:27:04,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:27:04,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:27:04,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:05,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:05,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:07,485 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:07,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:07,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:09,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:11,255 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:27:11,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:11,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:13,487 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:14,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:14,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:14,615 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:16,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:16,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:19,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:27:19,130 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:27:19,131 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:27:19,133 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:27:19,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:19,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:19,489 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:20,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:21,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:21,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:23,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:23,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:25,491 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:25,747 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:26,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:26,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:28,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:28,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:30,748 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:31,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:31,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:31,493 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:33,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:33,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:34,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:27:34,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:27:34,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:27:36,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:36,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:36,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:37,495 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:38,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:38,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:41,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:41,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:41,879 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:43,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:43,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:43,497 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:46,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:46,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:46,958 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:49,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:27:49,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:27:49,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:27:49,258 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:27:49,698 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:27:49,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:49,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:51,500 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:52,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:52,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:52,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:54,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:54,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:57,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:27:57,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:57,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:58,221 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:27:59,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:27:59,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:02,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:02,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:03,290 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:03,505 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:04,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:28:04,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:28:04,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:28:04,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:04,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:07,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:28:07,418 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:28:07,419 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:28:07,421 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:28:07,506 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:28:07,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:07,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:08,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:09,507 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:09,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:09,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:12,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:12,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:14,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:15,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:15,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:15,510 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:18,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:18,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:19,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:28:19,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:28:19,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:28:19,348 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:28:19,490 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:20,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:20,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:21,512 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:22,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:22,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:24,490 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:24,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:25,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:27,515 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:28,533 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:28:28,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:28,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:29,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:31,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:31,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:33,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:33,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:33,518 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:34,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:28:34,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:28:34,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:28:35,345 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:35,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:35,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:38,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:38,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:39,520 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:40,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:40,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:40,674 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:43,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:43,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:45,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:45,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:45,522 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:45,732 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:48,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:48,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:49,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:28:49,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:28:49,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:28:49,315 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:28:50,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:50,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:50,886 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:51,524 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:53,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:53,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:55,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:55,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:55,886 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:28:57,526 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:28:58,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:28:58,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:00,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:00,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:00,955 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:03,529 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:03,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:03,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:04,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:29:04,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:29:04,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:29:06,344 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:07,139 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:29:07,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:07,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:09,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:09,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:09,531 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:12,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:12,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:12,085 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:14,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:14,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:15,533 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:17,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:17,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:17,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:19,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:19,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:29:19,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:29:19,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:29:19,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:20,751 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:29:21,535 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:22,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:22,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:22,752 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:24,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:24,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:26,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:26,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:27,537 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:28,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:29,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:29,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:31,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:31,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:33,338 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:33,539 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:34,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:29:34,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:29:34,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:29:34,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:34,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:36,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:36,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:38,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:39,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:39,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:39,542 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:41,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:41,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:43,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:44,943 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:29:44,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:44,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:45,545 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:47,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:47,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:48,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:49,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:29:49,129 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:29:49,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:29:49,265 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:29:49,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:49,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:51,547 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:52,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:52,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:53,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:54,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:54,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:57,549 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:29:57,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:57,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:58,576 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:29:59,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:29:59,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:01,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:01,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:03,551 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:03,645 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:04,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:30:04,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:30:04,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:30:04,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:04,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:06,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:06,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:08,694 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:09,554 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:09,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:09,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:11,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:11,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:13,695 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:14,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:14,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:15,555 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:16,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:16,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:18,759 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:19,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:30:19,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:30:19,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:30:19,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:19,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:20,265 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:30:21,558 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:23,396 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:30:23,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:23,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:23,832 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:26,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:26,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:27,560 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:28,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:28,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:28,877 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:30,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:30,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:33,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:33,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:33,562 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:33,932 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:34,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:30:34,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:30:34,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:30:35,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:35,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:38,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:38,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:38,981 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:39,052 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:30:39,054 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:30:39,054 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:30:39,054 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:30:39,565 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:30:39,566 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:40,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:40,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:41,566 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:43,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:43,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:44,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:45,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:45,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:45,568 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:48,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:48,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:49,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:30:49,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:30:49,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:30:49,575 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:50,322 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:30:50,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:50,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:50,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:51,575 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:53,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:53,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:55,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:55,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:55,577 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:55,839 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:30:57,577 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:30:58,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:30:58,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:01,423 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:02,145 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:31:02,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:02,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:03,579 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:04,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:31:04,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:31:04,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:31:04,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:04,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:06,725 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:06,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:06,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:07,580 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:08,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:08,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:11,725 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:11,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:11,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:13,582 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:13,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:13,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:16,794 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:16,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:16,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:18,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:18,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:19,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:31:19,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:31:19,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:31:19,347 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:31:19,584 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:21,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:21,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:22,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:23,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:23,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:25,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:25,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:25,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:27,907 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:28,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:28,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:31,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:31,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:31,588 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:32,981 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:34,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:34,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:34,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:31:34,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:31:34,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:31:36,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:36,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:37,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:38,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:39,657 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:31:39,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:39,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:42,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:42,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:43,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:44,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:44,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:45,592 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:47,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:47,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:48,314 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:49,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:31:49,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:31:49,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:31:49,307 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:31:49,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:49,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:51,594 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:52,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:52,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:53,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:54,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:54,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:57,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:57,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:57,596 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:31:58,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:31:59,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:31:59,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:02,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:02,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:03,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:03,598 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:04,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:32:04,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:32:04,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:32:04,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:04,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:06,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:06,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:08,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:09,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:09,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:09,600 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:11,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:11,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:13,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:14,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:14,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:15,603 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:17,983 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:32:18,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:18,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:18,602 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:19,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:32:19,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:32:19,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:32:20,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:20,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:20,887 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:32:21,605 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:22,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:22,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:23,888 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:24,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:24,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:27,607 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:27,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:27,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:29,738 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:29,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:29,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:31,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:31,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:33,609 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:34,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:32:34,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:32:34,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:32:34,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:34,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:35,349 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:36,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:36,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:39,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:39,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:39,611 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:40,860 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:41,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:41,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:44,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:44,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:45,613 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:45,903 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:46,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:46,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:49,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:32:49,131 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:32:49,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:32:49,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:49,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:50,512 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:32:51,514 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:51,615 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:51,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:51,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:55,145 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:32:55,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:55,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:56,104 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:32:56,106 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:32:56,106 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:32:56,107 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:32:56,616 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:32:57,107 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:32:57,617 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:32:58,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:32:58,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:00,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:00,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:02,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:02,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:03,088 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:03,618 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:04,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:33:04,130 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:33:04,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:33:05,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:05,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:08,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:08,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:08,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:09,620 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:10,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:10,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:13,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:13,133 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:13,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:15,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:15,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:15,622 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:18,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:18,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:18,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:19,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:33:19,131 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:33:19,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:33:20,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:20,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:20,580 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:33:21,624 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:23,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:23,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:23,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:25,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:25,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:27,626 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:28,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:28,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:29,309 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:30,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:30,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:33,788 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:33:33,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:33,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:34,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:33:34,131 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:33:34,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:33:34,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:34,628 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:36,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:36,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:38,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:38,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:39,424 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:40,630 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:41,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:41,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:43,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:43,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:44,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:46,632 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:46,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:46,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:48,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:48,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:49,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:33:49,131 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:33:49,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:33:49,351 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:33:50,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:50,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:51,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:52,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:53,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:53,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:55,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:33:56,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:56,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:58,636 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:33:58,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:33:58,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:00,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:01,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:01,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:03,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:03,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:04,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:34:04,132 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:34:04,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:34:04,638 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:06,337 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:06,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:06,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:08,639 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:08,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:08,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:10,640 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:11,870 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:12,291 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:34:12,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:12,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:14,641 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:15,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:15,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:16,642 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:17,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:17,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:17,601 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:19,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:34:19,132 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:34:19,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:34:19,289 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:34:20,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:20,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:20,643 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:22,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:22,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:23,292 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:25,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:25,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:26,645 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:27,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:27,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:28,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:30,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:30,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:30,647 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:32,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:32,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:34,010 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:34,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:34:34,132 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:34:34,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:34:34,648 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:35,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:35,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:37,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:37,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:39,917 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:40,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:40,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:40,650 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:42,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:42,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:45,056 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:45,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:45,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:46,652 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:47,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:47,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:49,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:34:49,133 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:34:49,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:34:49,310 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:34:50,335 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:50,563 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:34:50,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:50,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:50,654 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:53,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:53,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:55,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:55,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:56,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:34:56,656 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:34:58,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:34:58,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:00,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:00,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:01,894 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:02,658 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:03,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:03,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:04,133 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:35:04,134 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:35:04,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:35:05,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:05,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:05,847 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:35:05,848 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:35:05,849 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:35:05,849 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:35:06,660 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:35:07,660 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:07,850 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:08,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:08,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:08,661 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:10,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:10,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:13,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:13,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:13,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:14,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:15,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:15,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:18,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:18,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:18,665 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:19,133 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:35:19,134 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:35:19,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:35:20,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:20,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:20,455 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:35:20,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:22,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:22,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:24,667 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:24,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:24,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:26,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:28,199 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:35:28,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:28,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:28,669 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:31,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:31,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:32,306 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:33,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:33,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:34,133 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:35:34,134 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:35:34,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:35:34,671 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:35,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:35,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:37,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:37,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:37,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:40,673 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:40,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:40,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:42,605 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:42,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:42,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:44,674 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:45,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:45,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:47,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:47,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:48,584 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:49,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:35:49,134 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:35:49,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:35:50,234 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:35:50,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:50,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:52,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:52,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:52,676 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:54,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:35:55,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:55,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:56,678 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:35:57,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:57,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:35:59,950 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:00,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:00,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:02,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:02,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:02,680 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:04,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:36:04,134 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:36:04,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:36:05,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:05,969 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:36:05,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:06,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:08,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:08,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:08,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:10,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:10,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:10,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:12,683 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:13,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:13,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:15,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:15,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:15,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:18,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:18,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:18,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:19,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:36:19,134 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:36:19,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:36:20,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:20,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:20,923 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:36:20,926 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:23,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:23,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:24,687 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:25,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:25,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:26,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:28,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:28,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:28,689 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:30,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:30,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:32,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:33,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:33,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:34,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:36:34,134 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:36:34,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:36:34,691 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:35,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:35,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:37,357 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:38,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:38,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:40,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:40,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:40,694 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:43,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:43,616 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:36:43,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:43,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:46,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:46,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:46,696 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:48,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:48,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:48,813 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:49,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:36:49,134 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:36:49,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:36:49,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:36:51,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:51,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:52,698 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:53,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:53,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:54,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:36:55,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:55,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:56,699 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:36:58,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:58,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:36:59,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:01,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:01,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:02,702 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:03,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:03,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:04,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:37:04,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:37:04,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:37:05,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:05,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:05,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:07,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:07,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:08,704 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:10,819 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:10,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:10,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:12,705 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:12,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:12,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:15,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:15,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:16,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:17,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:17,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:18,707 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:19,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:37:19,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:37:19,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:37:19,290 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:37:21,071 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:37:21,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:21,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:21,291 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:22,107 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:37:22,108 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:37:22,109 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:37:22,111 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:37:22,709 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:37:23,710 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:23,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:23,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:24,710 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:25,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:25,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:27,113 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:28,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:28,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:30,712 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:30,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:30,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:32,933 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:33,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:33,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:34,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:37:34,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:37:34,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:37:34,714 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:35,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:35,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:38,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:38,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:38,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:40,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:40,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:40,718 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:43,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:43,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:43,755 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:45,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:45,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:46,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:48,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:48,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:49,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:37:49,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:37:49,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:37:49,324 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:37:49,324 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:50,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:50,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:50,721 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:53,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:53,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:54,514 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:37:55,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:55,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:56,723 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:37:59,264 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:37:59,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:37:59,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:00,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:01,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:01,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:02,726 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:04,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:38:04,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:38:04,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:38:04,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:04,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:06,314 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:06,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:06,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:08,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:09,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:09,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:11,495 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:11,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:11,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:14,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:14,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:14,730 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:16,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:16,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:17,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:18,731 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:19,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:38:19,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:38:19,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:38:19,352 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:38:19,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:19,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:21,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:21,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:22,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:24,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:24,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:24,734 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:26,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:26,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:28,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:29,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:29,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:30,736 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:31,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:31,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:33,732 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:34,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:38:34,136 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:38:34,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:38:34,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:34,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:36,738 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:38,067 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:38:38,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:38,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:39,340 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:40,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:40,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:42,740 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:42,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:43,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:44,913 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:45,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:45,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:46,742 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:47,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:47,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:49,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:38:49,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:38:49,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:38:49,364 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:38:50,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:50,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:50,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:52,743 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:38:52,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:52,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:55,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:38:55,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:55,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:57,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:57,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:38:58,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:00,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:00,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:01,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:02,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:02,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:02,746 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:04,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:39:04,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:39:04,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:39:05,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:05,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:06,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:07,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:07,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:08,748 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:10,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:10,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:12,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:12,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:12,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:14,751 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:15,787 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:39:15,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:15,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:17,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:18,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:18,752 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:18,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:19,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:39:19,135 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:39:19,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:39:19,329 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:39:20,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:20,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:22,330 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:23,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:23,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:24,754 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:25,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:25,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:27,686 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:28,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:28,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:30,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:30,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:30,756 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:33,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:33,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:33,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:34,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:39:34,136 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:39:34,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:39:35,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:35,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:36,758 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:38,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:38,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:39,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:39,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:39:39,475 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:39:39,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:39:39,476 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:39:39,759 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:39:39,759 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:40,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:40,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:40,760 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:43,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:43,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:44,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:45,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:45,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:46,761 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:48,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:48,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:49,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:39:49,136 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:39:49,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:39:49,295 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:39:50,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:50,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:50,296 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:52,763 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:53,632 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:39:53,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:53,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:55,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:39:56,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:56,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:56,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:39:58,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:39:58,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:00,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:01,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:01,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:02,767 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:03,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:03,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:04,136 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:40:04,136 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:40:04,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:40:06,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:06,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:06,362 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:08,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:08,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:08,769 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:10,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:10,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:11,881 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:12,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:13,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:13,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:16,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:16,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:16,894 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:18,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:18,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:18,773 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:19,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:40:19,136 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:40:19,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:40:19,360 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:40:21,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:21,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:22,362 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:23,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:23,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:24,775 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:26,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:26,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:27,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:28,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:28,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:30,777 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:31,963 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:40:31,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:32,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:33,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:34,136 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:40:34,136 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:40:34,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:40:34,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:34,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:36,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:36,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:36,779 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:39,090 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:39,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:39,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:40,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:41,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:41,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:44,093 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:44,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:44,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:46,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:46,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:46,783 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:49,137 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:40:49,137 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:40:49,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:40:49,292 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:40:49,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:49,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:49,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:51,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:51,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:52,785 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:54,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:54,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:55,025 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:40:56,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:56,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:56,786 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:40:59,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:40:59,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:00,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:01,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:01,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:02,788 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:04,137 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:41:04,138 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:41:04,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:41:04,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:04,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:06,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:06,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:06,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:08,790 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:09,832 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:41:09,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:09,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:11,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:12,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:12,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:14,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:14,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:14,792 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:17,363 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:17,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:17,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:18,793 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:19,137 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:41:19,137 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:41:19,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:41:19,357 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:41:19,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:19,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:22,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:22,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:23,085 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:24,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:24,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:24,795 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:27,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:27,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:28,085 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:29,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:29,419 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:30,797 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:32,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:32,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:33,890 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:34,137 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:41:34,138 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:41:34,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:41:34,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:34,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:34,798 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:37,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:37,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:39,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:39,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:39,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:40,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:42,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:42,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:44,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:44,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:44,667 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:46,803 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:47,708 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:41:47,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:47,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:49,137 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:41:49,138 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:41:49,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:41:50,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:50,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:51,187 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:41:51,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:52,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:52,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:52,804 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:55,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:55,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:56,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:41:56,497 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:41:56,498 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:41:56,499 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:41:56,499 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:41:56,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:41:57,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:57,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:41:57,806 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:41:58,806 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:00,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:00,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:01,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:02,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:02,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:02,808 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:04,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:42:04,138 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:42:04,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:42:05,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:05,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:07,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:07,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:07,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:08,811 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:10,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:10,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:12,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:12,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:12,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:14,813 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:15,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:15,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:17,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:17,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:17,737 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:19,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:42:19,138 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:42:19,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:42:19,354 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:42:20,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:20,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:20,815 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:22,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:22,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:23,356 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:25,431 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:42:25,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:25,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:26,817 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:28,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:28,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:28,669 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:30,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:30,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:30,818 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:33,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:33,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:34,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:42:34,138 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:42:34,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:42:34,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:35,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:35,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:36,820 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:38,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:38,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:39,499 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:40,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:40,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:42,822 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:43,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:43,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:44,844 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:45,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:45,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:48,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:48,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:48,824 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:49,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:42:49,138 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:42:49,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:42:49,305 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:42:50,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:50,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:50,306 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:53,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:53,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:54,826 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:55,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:55,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:56,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:42:57,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:57,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:58,827 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:42:59,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:42:59,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:02,030 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:03,294 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:43:03,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:03,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:04,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:43:04,138 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:43:04,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:43:04,830 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:06,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:06,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:07,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:08,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:08,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:10,831 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:11,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:11,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:12,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:13,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:13,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:14,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:15,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:15,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:17,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:17,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:18,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:19,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:43:19,139 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:43:19,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:43:19,302 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:43:20,834 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:20,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:20,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:22,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:22,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:23,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:25,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:25,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:26,837 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:27,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:27,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:29,205 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:30,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:30,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:30,838 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:32,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:32,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:34,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:43:34,139 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:43:34,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:43:34,309 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:35,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:35,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:36,841 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:37,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:37,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:39,938 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:41,122 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:43:41,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:41,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:42,843 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:43,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:43,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:45,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:45,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:45,935 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:48,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:48,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:48,846 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:49,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:43:49,139 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:43:49,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:43:49,342 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:43:50,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:50,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:51,343 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:52,847 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:53,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:53,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:55,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:55,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:56,615 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:43:58,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:43:58,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:43:58,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:00,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:00,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:02,564 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:03,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:03,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:04,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:44:04,139 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:44:04,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:44:04,852 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:05,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:05,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:08,357 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:08,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:09,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:10,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:12,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:12,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:13,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:14,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:14,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:44:14,366 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:44:14,366 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:44:14,367 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:44:14,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:14,855 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:44:16,856 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:17,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:17,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:19,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:44:19,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:44:19,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:44:19,418 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:44:19,419 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:20,790 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:44:20,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:20,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:20,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:23,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:23,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:24,419 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:25,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:25,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:26,860 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:28,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:28,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:30,017 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:30,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:30,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:32,862 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:33,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:33,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:34,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:44:34,139 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:44:34,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:44:35,358 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:35,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:35,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:36,864 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:38,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:38,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:40,749 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:40,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:40,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:42,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:43,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:43,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:45,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:45,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:46,663 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:48,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:48,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:48,868 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:49,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:44:49,139 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:44:49,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:44:49,272 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:44:50,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:50,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:52,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:52,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:52,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:54,871 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:44:55,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:55,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:57,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:44:59,181 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:44:59,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:44:59,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:00,873 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:02,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:02,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:03,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:04,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:45:04,139 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:45:04,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:45:04,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:04,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:04,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:07,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:07,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:08,372 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:09,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:09,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:10,876 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:11,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:11,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:13,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:13,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:14,080 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:15,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:15,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:16,878 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:18,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:18,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:19,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:45:19,139 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:45:19,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:45:20,293 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:45:20,294 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:20,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:20,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:20,880 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:23,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:23,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:25,295 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:25,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:25,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:26,882 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:28,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:28,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:30,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:30,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:30,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:32,884 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:33,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:33,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:34,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:45:34,140 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:45:34,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:45:35,902 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:36,886 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:37,046 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:45:37,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:37,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:40,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:40,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:40,903 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:42,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:42,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:42,888 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:45,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:45,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:46,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:47,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:47,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:48,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:49,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:45:49,140 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:45:49,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:45:49,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:49,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:50,914 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:45:51,915 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:51,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:51,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:52,892 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:45:54,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:54,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:56,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:56,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:56,919 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:45:59,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:59,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:45:59,894 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:01,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:01,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:01,920 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:04,140 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:46:04,140 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:46:04,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:46:04,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:04,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:06,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:06,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:07,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:09,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:09,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:11,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:11,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:12,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:14,979 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:46:15,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:15,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:15,899 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:17,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:17,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:18,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:19,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:46:19,140 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:46:19,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:46:19,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:19,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:20,560 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:46:22,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:22,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:23,561 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:23,902 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:24,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:24,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:27,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:27,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:27,903 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:28,641 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:29,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:29,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:32,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:32,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:33,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:34,140 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:46:34,141 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:46:34,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:46:34,290 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:34,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:34,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:37,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:37,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:37,907 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:39,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:39,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:39,831 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:41,096 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:46:41,097 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:46:41,097 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:46:41,098 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:46:41,909 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:46:42,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:42,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:42,909 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:43,910 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:44,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:44,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:45,099 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:47,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:47,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:49,140 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:46:49,141 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:46:49,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:46:49,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:49,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:49,377 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:46:49,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:50,378 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:52,611 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:46:52,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:52,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:53,913 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:46:55,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:55,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:55,761 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:46:57,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:57,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:59,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:59,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:46:59,915 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:01,675 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:02,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:02,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:04,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:47:04,141 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:47:04,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:47:04,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:04,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:05,917 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:07,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:07,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:07,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:09,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:09,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:09,918 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:12,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:12,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:12,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:14,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:14,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:15,920 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:17,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:17,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:18,333 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:19,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:47:19,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:47:19,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:47:19,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:19,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:20,607 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:47:21,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:21,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:23,609 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:23,923 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:23,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:23,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:27,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:27,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:27,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:29,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:30,895 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:47:30,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:30,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:32,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:32,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:33,927 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:34,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:47:34,141 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:47:34,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:47:34,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:35,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:35,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:37,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:37,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:37,928 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:39,954 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:40,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:40,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:42,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:42,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:43,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:44,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:44,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:45,852 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:47,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:47,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:49,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:47:49,141 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:47:49,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:47:49,933 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:50,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:50,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:50,698 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:47:51,699 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:52,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:52,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:54,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:54,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:55,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:47:57,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:57,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:57,629 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:47:59,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:59,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:47:59,936 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:01,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:01,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:03,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:04,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:48:04,141 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:48:04,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:48:04,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:04,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:05,937 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:08,448 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:48:08,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:08,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:09,292 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:10,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:10,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:11,940 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:13,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:13,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:14,326 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:15,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:15,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:15,941 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:17,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:17,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:19,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:48:19,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:48:19,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:48:19,324 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:48:20,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:20,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:20,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:21,943 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:22,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:22,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:24,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:25,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:25,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:27,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:27,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:27,945 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:29,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:29,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:31,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:31,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:31,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:33,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:34,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:48:34,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:48:34,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:48:34,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:34,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:36,369 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:36,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:36,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:38,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:38,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:39,949 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:41,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:41,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:42,008 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:43,950 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:45,464 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:48:45,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:45,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:47,008 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:48,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:48,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:49,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:48:49,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:48:49,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:48:49,356 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:48:49,952 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:50,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:50,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:52,357 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:53,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:53,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:55,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:55,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:55,954 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:48:57,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:57,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:48:57,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:48:58,582 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:48:58,584 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:48:58,584 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:48:58,585 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:48:58,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:48:59,956 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:00,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:00,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:00,956 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:02,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:02,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:03,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:04,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:49:04,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:49:04,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:49:04,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:04,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:05,958 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:07,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:07,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:09,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:09,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:09,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:11,960 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:12,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:12,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:14,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:14,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:14,732 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:17,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:17,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:17,961 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:19,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:49:19,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:49:19,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:49:19,336 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:49:19,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:19,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:20,337 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:23,103 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:49:23,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:23,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:23,964 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:26,115 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:26,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:26,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:27,965 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:28,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:28,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:31,116 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:31,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:31,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:33,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:33,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:33,966 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:34,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:49:34,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:49:34,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:49:36,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:36,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:36,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:38,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:38,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:39,968 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:41,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:41,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:41,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:43,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:43,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:45,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:45,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:45,970 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:47,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:48,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:48,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:49,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:49:49,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:49:49,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:49:49,972 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:50,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:50,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:50,642 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:49:52,643 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:49:52,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:52,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:55,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:55,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:55,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:49:57,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:57,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:49:57,944 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:01,127 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:50:01,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:01,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:01,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:03,867 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:04,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:04,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:04,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:50:04,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:50:04,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:50:06,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:06,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:07,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:08,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:08,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:09,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:11,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:11,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:11,980 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:13,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:13,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:14,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:15,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:15,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:17,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:17,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:17,982 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:19,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:50:19,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:50:19,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:50:20,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:20,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:20,510 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:50:20,512 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:22,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:22,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:23,986 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:25,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:25,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:26,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:27,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:27,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:29,989 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:30,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:30,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:32,345 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:32,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:32,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:34,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:50:34,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:50:34,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:50:34,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:34,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:35,991 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:38,323 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:50:38,334 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:38,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:38,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:41,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:41,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:41,993 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:43,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:43,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:43,943 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:46,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:46,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:47,995 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:48,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:48,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:49,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:50:49,143 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:50:49,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:50:50,804 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:50:50,804 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:51,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:51,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:53,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:53,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:53,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:56,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:56,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:56,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:50:57,999 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:50:58,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:50:58,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:01,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:01,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:01,809 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:03,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:03,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:04,001 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:04,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:51:04,142 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:51:04,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:51:06,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:06,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:07,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:08,002 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:08,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:08,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:11,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:11,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:13,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:13,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:13,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:14,004 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:16,710 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:51:16,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:16,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:16,920 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:51:16,921 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:51:16,921 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:51:16,921 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:51:17,006 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:51:18,922 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:19,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:19,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:51:19,143 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:51:19,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:51:19,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:19,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:20,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:20,604 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:51:21,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:21,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:24,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:24,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:24,816 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:26,009 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:26,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:26,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:29,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:29,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:30,010 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:30,635 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:31,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:31,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:34,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:51:34,143 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:51:34,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:51:34,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:34,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:36,013 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:36,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:36,349 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:36,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:39,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:39,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:41,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:41,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:41,692 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:42,015 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:44,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:44,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:46,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:46,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:46,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:46,760 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:49,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:51:49,143 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:51:49,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:51:49,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:49,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:49,267 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:51:51,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:51,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:52,018 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:52,702 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:54,751 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:51:54,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:54,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:57,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:57,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:58,020 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:51:58,384 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:51:59,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:51:59,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:02,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:02,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:03,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:04,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:04,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:52:04,143 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:52:04,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:52:04,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:04,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:07,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:07,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:09,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:09,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:09,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:10,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:12,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:12,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:14,026 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:14,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:14,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:14,692 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:17,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:17,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:19,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:52:19,143 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:52:19,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:52:19,330 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:52:19,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:19,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:20,029 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:20,331 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:22,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:22,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:24,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:24,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:26,031 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:26,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:27,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:27,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:29,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:29,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:31,198 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:32,033 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:32,871 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:52:32,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:32,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:34,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:52:34,143 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:52:34,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:52:35,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:35,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:37,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:37,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:37,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:38,035 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:40,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:40,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:42,036 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:42,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:42,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:43,034 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:45,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:45,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:47,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:47,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:48,038 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:48,958 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:49,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:52:49,143 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:52:49,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:52:49,286 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:52:50,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:50,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:52,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:52,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:54,041 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:52:54,769 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:52:55,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:55,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:57,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:52:57,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:00,043 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:00,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:00,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:00,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:02,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:02,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:04,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:53:04,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:53:04,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:53:05,363 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:05,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:05,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:06,045 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:07,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:07,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:10,046 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:10,698 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:11,111 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:53:11,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:11,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:13,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:13,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:16,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:16,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:16,048 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:16,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:18,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:18,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:19,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:53:19,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:53:19,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:53:19,314 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:53:20,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:21,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:22,050 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:22,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:23,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:23,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:26,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:26,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:26,051 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:28,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:28,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:28,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:30,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:30,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:32,054 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:33,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:33,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:34,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:53:34,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:53:34,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:53:34,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:35,298 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:53:35,299 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:53:35,299 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:53:35,300 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:53:35,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:35,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:36,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:53:37,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:38,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:38,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:38,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:39,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:40,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:40,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:43,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:43,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:44,057 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:45,069 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:45,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:45,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:48,058 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:49,071 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:53:49,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:49,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:49,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:53:49,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:53:49,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:53:49,371 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:53:50,372 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:51,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:51,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:53,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:53,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:54,060 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:53:55,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:53:56,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:56,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:58,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:53:58,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:00,062 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:01,769 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:01,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:01,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:03,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:03,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:04,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:54:04,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:54:04,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:54:06,064 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:06,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:06,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:07,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:08,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:08,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:11,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:11,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:12,067 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:13,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:13,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:13,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:16,068 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:16,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:16,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:18,660 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:18,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:18,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:19,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:54:19,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:54:19,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:54:19,319 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:54:21,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:21,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:22,069 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:23,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:23,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:23,874 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:26,873 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:54:26,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:26,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:28,071 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:29,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:29,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:29,771 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:31,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:31,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:32,072 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:34,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:54:34,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:54:34,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:54:34,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:34,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:35,324 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:36,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:36,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:38,074 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:39,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:39,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:40,726 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:41,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:41,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:44,075 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:44,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:44,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:46,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:46,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:46,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:49,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:54:49,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:54:49,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:54:49,286 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:54:49,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:49,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:50,078 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:51,287 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:51,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:51,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:54,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:54,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:56,079 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:54:56,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:56,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:56,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:54:59,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:54:59,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:00,081 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:01,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:01,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:02,312 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:04,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:55:04,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:55:04,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:55:04,720 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:55:04,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:04,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:06,086 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:07,325 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:07,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:07,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:09,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:09,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:12,087 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:12,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:12,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:12,878 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:14,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:14,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:17,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:17,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:18,090 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:18,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:19,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:55:19,144 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:55:19,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:55:19,291 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:55:19,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:19,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:22,091 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:22,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:22,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:24,113 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:24,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:24,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:27,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:27,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:28,092 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:29,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:29,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:30,062 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:32,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:32,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:34,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:34,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:55:34,145 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:55:34,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:55:34,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:34,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:35,300 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:36,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:36,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:38,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:39,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:40,097 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:40,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:42,366 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:55:42,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:42,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:44,098 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:45,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:45,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:45,814 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:47,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:47,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:49,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:55:49,145 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:55:49,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:55:49,363 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:55:49,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:49,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:50,100 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:51,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:51,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:51,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:53,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:55:53,844 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:55:53,844 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:55:53,844 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:55:54,101 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:55:54,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:54,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:55,101 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:56,101 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:55:56,845 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:55:56,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:56,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:55:59,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:00,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:02,103 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:02,659 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:03,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:03,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:04,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:56:04,145 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:56:04,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:56:05,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:06,105 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:07,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:07,939 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:10,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:10,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:12,106 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:13,424 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:13,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:13,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:15,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:15,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:18,108 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:18,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:18,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:19,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:56:19,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:56:19,237 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:56:19,373 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:56:19,374 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:21,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:21,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:22,109 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:25,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:25,524 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:56:25,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:25,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:28,112 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:28,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:28,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:30,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:30,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:31,002 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:33,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:33,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:34,114 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:34,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:56:34,145 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:56:34,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:56:35,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:35,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:36,309 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:38,115 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:38,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:38,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:40,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:40,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:41,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:43,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:43,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:44,117 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:45,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:45,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:47,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:48,118 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:48,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:48,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:49,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:56:49,145 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:56:49,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:56:50,120 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:50,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:50,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:50,769 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:56:52,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:53,120 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:53,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:53,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:55,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:55,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:57,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:56:58,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:56:58,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:58,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:56:59,123 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:01,124 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:02,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:02,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:03,610 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:04,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:57:04,145 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:57:04,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:57:05,125 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:05,869 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:57:05,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:05,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:08,636 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:08,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:08,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:09,126 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:10,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:10,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:13,636 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:13,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:13,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:15,128 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:15,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:15,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:17,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:17,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:18,674 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:19,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:57:19,145 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:57:19,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:57:19,317 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:57:20,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:20,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:21,130 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:22,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:22,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:23,727 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:24,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:24,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:27,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:27,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:27,133 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:28,776 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:29,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:29,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:32,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:32,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:33,134 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:33,812 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:34,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:57:34,145 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:57:34,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:57:35,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:35,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:37,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:37,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:38,879 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:39,137 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:40,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:40,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:43,757 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:57:43,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:43,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:43,880 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:45,138 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:45,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:45,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:48,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:48,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:48,931 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:49,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:57:49,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:57:49,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:57:49,347 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:57:50,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:50,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:51,140 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:52,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:52,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:53,998 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:55,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:55,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:56,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 19:57:56,127 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 19:57:56,127 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 19:57:56,128 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 19:57:56,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 19:57:57,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:57,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:57,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:57:58,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:57:59,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:57:59,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:00,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:02,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:02,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:03,144 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:04,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:58:04,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:58:04,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:58:04,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:04,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:04,364 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:07,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:07,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:09,146 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:09,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:09,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:10,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:12,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:12,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:14,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:14,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:15,148 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:15,206 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:17,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:17,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:19,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:58:19,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:58:19,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:58:19,343 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:58:20,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:21,021 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:58:21,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:21,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:21,150 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:23,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:23,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:25,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:26,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:26,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:27,153 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:28,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:28,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:30,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:30,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:30,308 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:33,154 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:33,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:33,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:34,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:58:34,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:58:34,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:58:35,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:35,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:35,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:37,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:37,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:40,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:40,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:40,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:41,157 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:42,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:42,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:45,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:45,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:45,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:47,159 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:47,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:47,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:49,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:58:49,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:58:49,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:58:49,299 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:58:50,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:50,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:50,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:52,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:52,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:53,161 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:58:54,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:54,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:55,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:58:58,413 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:58:58,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:58,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:58:59,163 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:00,612 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:01,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:01,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:03,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:03,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:04,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:59:04,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:59:04,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:59:05,165 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:06,309 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:06,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:06,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:08,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:08,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:11,167 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:11,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:11,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:11,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:13,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:13,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:16,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:16,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:16,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:17,169 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:18,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:18,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:19,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:59:19,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:59:19,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:59:19,345 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:59:21,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:21,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:21,863 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:23,171 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:23,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:23,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:26,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:26,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:26,864 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:28,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:28,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:29,173 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:31,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:31,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:31,911 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:33,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:33,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:34,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:59:34,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:59:34,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:59:35,175 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:37,253 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 19:59:37,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:37,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:37,382 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:40,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:40,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:41,177 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:42,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:42,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:43,045 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:44,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:44,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:46,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:46,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:47,179 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:48,103 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:49,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 19:59:49,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 19:59:49,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 19:59:49,293 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 19:59:49,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:49,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:52,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:52,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:53,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:53,181 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:55,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:55,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:57,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:57,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 19:59:58,210 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 19:59:59,183 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 19:59:59,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:00,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:02,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:02,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:03,211 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:04,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:00:04,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:00:04,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:00:05,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:05,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:05,185 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:07,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:07,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:08,703 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:09,187 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:10,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:10,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:11,187 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:12,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:12,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:14,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:15,189 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:15,694 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:00:15,943 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:00:15,945 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:00:15,945 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:00:15,946 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:00:16,189 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:00:16,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:16,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:17,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:18,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:18,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:18,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:19,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:00:19,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:00:19,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:00:21,090 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:00:21,091 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:21,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:21,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:23,191 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:23,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:23,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:26,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:26,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:26,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:27,193 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:29,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:29,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:31,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:31,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:31,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:33,194 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:34,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:00:34,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:00:34,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:00:34,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:34,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:36,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:36,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:36,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:38,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:38,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:39,197 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:41,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:41,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:42,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:43,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:43,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:45,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:46,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:46,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:47,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:48,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:48,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:49,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:00:49,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:00:49,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:00:50,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:50,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:50,935 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:00:52,936 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:53,202 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:53,999 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:00:54,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:54,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:57,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:57,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:57,203 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:00:58,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:00:59,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:00:59,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:01,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:01,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:03,205 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:03,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:04,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:01:04,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:01:04,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:01:04,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:04,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:06,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:06,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:08,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:09,207 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:09,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:09,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:11,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:11,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:13,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:14,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:14,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:15,209 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:16,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:16,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:18,659 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:19,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:01:19,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:01:19,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:01:19,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:19,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:20,440 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:01:21,211 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:21,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:21,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:24,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:24,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:24,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:26,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:26,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:27,213 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:28,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:28,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:29,797 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:32,116 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:01:32,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:32,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:33,215 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:34,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:01:34,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:01:34,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:01:34,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:34,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:35,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:36,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:36,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:39,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:39,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:39,217 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:40,933 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:41,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:41,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:43,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:43,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:45,219 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:45,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:45,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:46,001 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:48,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:48,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:49,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:01:49,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:01:49,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:01:49,288 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:01:50,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:50,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:51,069 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:51,221 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:53,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:53,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:55,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:55,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:56,070 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:01:57,222 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:01:57,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:01:58,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:01,133 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:01,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:01,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:03,224 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:03,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:03,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:04,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:02:04,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:02:04,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:02:05,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:05,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:06,317 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:08,739 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:02:08,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:08,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:09,226 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:11,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:11,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:12,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:13,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:13,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:15,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:15,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:15,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:17,341 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:18,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:18,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:19,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:02:19,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:02:19,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:02:19,264 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:02:20,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:20,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:21,230 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:22,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:22,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:22,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:24,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:24,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:26,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:26,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:27,232 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:27,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:28,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:28,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:31,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:31,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:32,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:33,235 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:33,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:33,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:34,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:02:34,148 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:02:34,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:02:36,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:36,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:38,330 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:38,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:38,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:39,237 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:41,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:41,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:43,633 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:44,785 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:02:44,786 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:02:44,786 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:02:44,788 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:02:45,240 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:02:45,633 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:02:45,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:45,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:46,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:47,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:47,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:47,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:48,789 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:49,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:02:49,147 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:02:49,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:02:49,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:49,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:50,723 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:02:53,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:53,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:53,243 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:02:53,792 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:55,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:55,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:58,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:58,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:02:58,896 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:02:59,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:00,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:00,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:03,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:03,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:03,961 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:04,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:03:04,148 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:03:04,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:03:05,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:05,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:05,247 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:08,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:08,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:09,036 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:10,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:10,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:11,249 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:12,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:12,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:14,036 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:14,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:14,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:17,251 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:17,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:17,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:19,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:19,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:03:19,148 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:03:19,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:03:19,316 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:03:19,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:19,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:23,161 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:03:23,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:23,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:23,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:24,214 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:26,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:26,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:28,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:28,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:29,255 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:29,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:31,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:31,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:33,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:33,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:34,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:03:34,148 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:03:34,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:03:34,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:35,258 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:35,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:35,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:37,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:37,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:39,355 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:39,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:39,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:41,260 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:42,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:42,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:44,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:44,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:44,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:47,262 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:47,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:47,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:49,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:03:49,148 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:03:49,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:03:49,331 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:03:49,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:49,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:50,332 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:52,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:52,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:53,264 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:03:54,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:54,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:55,637 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:03:56,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:56,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:03:59,267 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:00,560 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:04:00,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:00,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:00,719 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:03,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:03,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:04,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:04:04,148 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:04:04,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:04:05,269 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:05,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:05,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:06,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:07,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:07,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:10,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:10,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:11,272 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:11,909 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:12,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:12,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:15,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:15,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:17,008 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:17,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:17,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:17,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:19,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:04:19,148 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:04:19,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:04:19,369 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:04:20,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:20,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:22,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:22,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:22,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:23,276 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:24,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:25,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:27,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:28,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:28,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:29,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:30,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:30,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:32,220 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:32,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:32,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:34,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:04:34,149 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:04:34,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:04:35,280 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:35,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:35,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:37,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:39,094 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:04:39,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:39,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:41,283 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:41,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:41,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:42,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:44,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:44,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:46,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:46,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:47,285 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:47,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:48,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:48,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:49,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:04:49,148 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:04:49,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:04:50,417 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:04:51,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:51,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:52,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:53,287 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:04:53,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:53,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:55,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:55,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:57,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:04:58,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:58,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:04:59,289 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:00,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:00,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:02,749 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:03,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:03,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:04,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:05:04,149 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:05:04,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:05:05,292 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:05,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:05,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:07,840 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:08,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:08,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:11,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:11,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:12,948 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:13,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:13,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:13,295 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:16,902 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:05:16,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:16,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:17,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:05:17,120 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:05:17,120 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:05:17,122 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:05:17,296 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:05:18,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:18,297 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:18,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:18,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:19,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:05:19,149 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:05:19,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:05:19,297 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:19,344 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:05:21,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:22,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:23,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:24,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:24,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:25,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:27,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:27,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:28,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:29,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:29,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:31,301 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:32,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:32,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:33,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:34,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:05:34,149 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:05:34,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:05:34,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:34,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:36,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:36,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:37,303 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:38,355 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:39,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:39,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:41,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:41,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:43,305 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:43,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:43,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:43,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:46,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:46,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:48,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:48,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:48,556 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:49,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:05:49,149 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:05:49,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:05:49,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:05:49,307 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:51,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:51,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:53,641 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:55,207 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:05:55,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:55,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:55,309 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:05:57,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:57,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:58,642 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:05:59,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:05:59,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:01,311 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:01,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:01,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:03,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:03,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:03,728 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:04,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:06:04,150 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:06:04,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:06:05,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:05,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:07,313 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:07,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:07,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:08,823 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:10,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:10,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:12,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:12,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:13,315 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:13,907 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:15,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:15,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:17,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:17,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:18,998 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:19,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:06:19,150 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:06:19,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:06:19,317 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:20,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:20,674 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:06:20,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:22,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:22,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:24,080 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:24,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:24,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:25,319 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:27,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:28,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:29,082 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:31,320 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:31,842 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:06:31,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:31,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:33,321 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:33,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:33,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:34,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:06:34,150 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:06:34,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:06:34,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:36,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:36,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:37,323 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:38,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:38,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:39,324 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:40,109 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:41,324 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:06:41,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:41,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:43,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:43,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:45,603 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:45,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:45,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:48,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:48,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:49,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:06:49,150 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:06:49,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:06:49,282 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:06:50,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:50,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:51,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:53,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:53,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:56,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:56,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:56,284 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:06:59,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:59,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:06:59,331 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:01,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:01,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:01,472 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:04,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:07:04,150 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:07:04,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:07:04,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:04,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:05,333 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:06,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:06,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:07,350 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:10,114 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:07:10,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:10,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:11,335 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:12,642 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:13,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:13,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:15,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:15,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:17,337 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:17,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:17,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:17,753 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:19,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:07:19,150 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:07:19,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:07:19,374 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:07:19,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:19,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:22,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:22,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:22,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:23,340 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:24,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:24,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:27,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:27,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:27,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:29,342 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:29,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:29,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:32,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:32,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:32,926 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:34,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:07:34,150 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:07:34,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:07:34,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:34,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:35,344 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:36,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:36,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:38,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:38,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:39,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:41,346 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:41,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:41,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:43,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:43,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:44,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:46,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:07:46,283 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:07:46,283 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:07:46,283 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:07:46,348 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:07:47,348 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:47,468 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:07:47,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:47,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:48,349 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:49,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:07:49,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:07:49,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:07:49,345 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:07:49,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:50,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:50,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:52,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:52,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:53,350 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:55,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:07:55,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:55,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:57,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:57,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:59,352 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:07:59,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:07:59,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:00,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:02,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:02,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:04,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:08:04,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:08:04,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:08:04,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:04,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:05,354 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:05,488 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:07,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:07,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:09,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:09,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:10,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:11,357 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:12,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:12,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:14,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:15,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:15,784 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:17,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:17,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:17,359 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:19,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:08:19,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:08:19,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:08:19,345 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:08:20,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:20,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:21,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:22,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:22,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:23,361 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:26,119 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:08:26,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:26,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:26,955 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:28,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:28,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:30,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:30,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:31,364 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:32,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:33,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:33,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:34,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:08:34,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:08:34,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:08:35,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:35,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:37,366 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:37,497 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:38,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:38,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:40,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:40,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:42,580 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:43,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:43,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:43,368 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:45,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:45,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:47,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:47,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:47,580 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:49,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:08:49,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:08:49,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:08:49,370 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:49,373 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:08:50,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:50,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:52,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:52,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:53,374 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:54,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:54,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:55,372 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:08:57,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:57,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:58,774 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:08:59,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:08:59,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:01,374 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:03,154 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:09:03,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:03,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:03,866 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:04,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:09:04,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:09:04,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:09:06,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:06,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:07,377 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:08,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:08,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:08,946 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:10,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:10,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:13,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:13,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:13,379 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:14,052 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:15,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:15,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:17,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:17,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:19,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:19,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:09:19,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:09:19,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:09:19,370 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:09:19,381 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:19,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:19,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:22,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:22,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:24,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:24,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:24,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:25,384 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:27,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:27,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:29,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:29,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:29,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:31,386 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:32,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:32,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:34,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:09:34,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:09:34,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:09:34,316 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:34,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:34,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:36,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:36,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:37,388 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:39,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:40,769 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:09:40,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:40,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:43,390 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:43,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:43,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:44,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:45,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:45,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:48,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:49,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:49,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:09:49,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:09:49,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:09:49,357 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:09:49,393 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:50,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:51,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:51,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:53,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:53,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:55,395 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:09:55,720 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:09:56,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:56,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:58,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:09:58,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:00,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:00,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:00,721 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:01,397 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:03,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:03,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:04,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:10:04,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:10:04,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:10:05,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:05,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:06,310 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:07,399 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:07,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:07,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:10,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:10,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:11,928 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:12,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:12,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:13,402 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:15,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:15,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:17,018 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:19,141 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:10:19,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:10:19,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:10:19,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:19,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:10:19,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:19,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:10:19,315 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:10:19,315 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:10:19,316 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:10:19,316 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:10:19,404 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:10:19,404 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:20,404 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:21,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:21,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:21,404 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:22,318 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:23,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:23,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:25,406 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:26,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:26,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:28,207 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:28,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:28,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:30,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:30,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:33,312 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:33,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:33,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:34,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:10:34,152 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:10:34,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:10:34,409 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:35,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:35,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:38,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:38,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:38,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:40,411 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:40,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:40,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:43,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:43,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:43,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:45,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:45,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:46,413 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:47,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:47,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:48,488 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:49,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:10:49,152 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:10:49,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:10:50,581 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:10:51,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:51,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:52,415 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:53,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:53,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:53,583 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:56,825 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:10:56,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:56,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:58,418 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:10:58,700 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:10:59,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:10:59,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:01,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:01,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:03,794 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:04,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:11:04,152 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:11:04,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:11:04,419 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:04,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:04,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:06,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:06,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:08,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:08,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:08,896 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:10,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:11,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:11,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:13,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:13,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:13,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:16,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:16,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:16,424 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:18,998 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:19,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:11:19,152 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:11:19,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:11:19,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:19,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:19,320 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:11:21,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:21,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:22,426 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:24,096 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:24,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:24,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:26,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:26,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:28,428 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:29,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:29,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:29,206 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:31,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:31,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:34,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:11:34,152 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:11:34,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:11:34,345 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:34,430 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:35,001 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:11:35,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:35,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:38,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:38,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:39,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:40,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:40,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:40,432 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:42,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:42,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:44,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:45,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:45,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:46,434 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:47,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:47,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:49,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:11:49,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:11:49,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:11:49,359 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:11:50,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:50,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:50,360 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:52,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:52,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:52,436 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:55,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:55,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:55,600 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:11:57,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:57,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:58,438 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:11:59,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:11:59,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:00,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:02,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:02,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:04,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:12:04,152 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:12:04,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:12:04,441 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:04,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:04,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:06,335 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:06,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:06,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:09,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:09,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:10,443 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:11,905 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:13,031 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:12:13,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:13,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:16,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:16,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:16,445 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:17,010 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:18,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:18,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:19,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:12:19,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:12:19,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:12:19,351 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:12:20,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:20,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:22,306 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:22,447 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:23,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:23,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:25,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:25,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:27,306 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:27,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:27,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:30,450 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:30,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:30,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:32,384 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:32,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:32,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:34,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:12:34,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:12:34,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:12:35,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:35,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:36,452 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:37,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:37,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:38,337 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:40,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:40,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:42,454 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:42,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:42,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:43,571 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:44,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:46,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:48,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:48,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:48,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:48,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:49,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:12:49,152 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:12:49,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:12:49,382 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:12:51,634 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:12:51,636 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:12:51,636 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:12:51,637 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:12:51,854 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:12:51,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:51,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:52,457 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:12:52,457 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:53,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:53,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:54,458 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:54,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:12:56,459 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:56,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:56,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:58,459 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:12:58,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:12:58,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:00,035 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:01,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:01,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:02,461 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:03,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:03,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:04,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:13:04,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:13:04,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:13:04,461 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:05,313 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:06,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:06,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:08,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:08,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:10,463 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:11,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:11,036 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:11,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:13,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:13,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:15,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:15,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:16,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:16,465 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:18,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:18,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:19,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:13:19,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:13:19,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:13:19,354 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:13:20,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:20,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:21,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:22,467 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:23,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:23,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:25,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:25,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:26,649 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:28,469 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:28,823 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:13:28,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:28,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:31,649 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:32,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:32,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:34,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:34,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:34,152 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:13:34,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:13:34,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:13:34,470 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:36,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:36,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:37,370 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:38,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:38,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:40,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:40,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:40,472 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:42,830 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:43,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:43,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:45,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:45,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:46,474 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:47,933 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:48,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:48,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:49,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:13:49,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:13:49,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:13:50,351 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:13:50,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:50,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:52,476 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:53,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:13:53,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:53,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:55,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:55,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:58,479 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:13:58,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:58,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:13:59,137 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:00,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:00,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:02,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:02,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:04,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:14:04,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:14:04,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:14:04,325 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:04,481 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:06,618 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:14:06,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:06,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:09,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:09,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:09,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:10,483 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:11,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:11,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:14,343 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:14,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:14,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:16,485 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:16,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:16,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:19,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:14:19,153 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:14:19,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:14:19,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:19,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:21,187 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:14:21,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:21,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:21,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:22,487 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:23,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:23,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:26,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:27,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:27,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:28,489 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:29,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:29,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:31,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:31,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:31,631 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:34,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:14:34,154 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:14:34,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:14:34,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:34,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:34,491 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:36,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:36,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:37,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:39,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:39,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:40,492 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:41,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:41,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:42,839 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:45,034 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:14:45,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:45,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:46,494 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:47,932 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:48,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:48,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:49,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:14:49,154 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:14:49,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:14:49,371 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:14:50,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:50,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:52,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:52,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:52,496 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:14:53,048 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:55,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:55,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:57,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:57,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:14:58,140 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:14:58,498 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:00,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:00,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:02,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:02,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:03,140 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:04,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:15:04,154 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:15:04,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:15:04,500 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:05,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:05,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:07,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:07,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:08,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:09,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:15:09,414 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:15:09,414 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:15:09,415 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:15:09,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:15:09,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:09,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:10,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:11,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:11,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:13,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:14,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:15,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:16,504 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:17,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:17,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:18,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:19,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:19,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:19,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:15:19,154 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:15:19,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:15:19,364 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:15:22,506 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:22,574 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:15:22,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:22,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:23,572 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:25,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:25,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:27,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:27,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:28,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:30,509 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:30,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:30,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:32,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:32,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:33,669 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:34,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:15:34,154 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:15:34,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:15:34,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:34,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:36,511 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:36,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:36,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:39,308 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:39,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:39,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:41,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:41,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:42,513 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:44,881 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:45,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:45,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:47,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:47,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:48,515 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:49,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:15:49,154 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:15:49,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:15:49,310 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:15:50,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:50,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:50,312 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:52,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:52,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:54,517 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:15:55,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:55,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:56,092 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:15:57,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:15:57,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:00,519 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:01,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:01,266 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:16:01,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:01,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:03,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:03,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:04,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:16:04,154 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:16:04,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:16:06,288 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:06,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:06,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:06,522 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:08,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:08,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:11,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:11,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:11,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:12,524 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:13,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:13,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:16,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:16,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:16,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:18,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:18,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:18,526 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:19,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:16:19,154 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:16:19,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:16:20,303 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:16:20,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:20,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:22,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:23,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:23,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:24,529 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:25,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:25,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:27,602 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:27,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:27,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:29,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:29,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:30,531 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:32,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:32,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:32,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:34,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:16:34,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:16:34,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:16:34,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:34,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:36,533 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:37,830 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:38,258 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:16:38,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:38,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:41,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:41,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:42,535 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:42,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:43,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:43,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:45,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:45,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:47,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:47,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:48,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:48,537 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:49,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:16:49,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:16:49,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:16:49,325 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:16:49,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:49,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:52,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:52,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:53,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:54,540 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:16:54,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:54,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:56,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:56,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:58,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:16:59,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:16:59,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:00,542 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:01,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:01,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:03,290 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:04,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:17:04,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:17:04,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:17:04,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:04,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:06,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:06,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:06,544 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:08,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:08,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:08,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:10,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:10,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:12,546 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:13,674 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:14,374 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:17:14,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:14,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:17,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:17,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:18,789 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:19,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:17:19,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:17:19,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:17:19,323 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:17:19,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:19,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:20,549 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:22,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:22,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:23,905 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:24,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:24,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:26,551 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:27,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:27,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:28,906 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:29,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:29,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:31,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:31,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:32,554 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:33,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:33,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:34,011 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:34,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:17:34,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:17:34,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:17:37,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:37,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:38,556 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:39,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:39,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:39,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:41,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:41,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:42,313 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:17:42,314 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:17:42,315 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:17:42,316 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:17:42,557 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:17:43,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:43,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:44,317 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:44,558 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:45,558 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:46,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:46,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:48,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:48,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:49,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:17:49,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:17:49,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:17:50,508 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:17:50,509 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:50,560 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:51,856 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:17:51,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:51,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:54,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:54,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:56,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:17:56,562 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:17:56,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:56,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:58,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:17:58,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:01,576 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:02,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:02,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:02,564 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:04,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:18:04,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:18:04,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:18:04,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:04,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:06,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:06,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:06,695 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:08,566 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:09,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:09,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:11,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:11,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:11,696 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:14,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:14,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:14,568 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:16,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:16,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:16,813 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:19,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:18:19,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:18:19,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:18:19,372 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:18:19,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:19,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:20,570 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:21,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:21,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:22,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:23,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:23,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:25,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:25,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:26,572 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:28,031 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:29,624 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:18:29,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:29,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:32,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:32,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:32,574 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:33,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:34,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:18:34,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:18:34,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:18:34,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:34,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:37,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:37,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:38,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:38,576 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:39,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:39,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:42,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:42,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:43,369 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:44,578 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:44,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:44,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:46,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:46,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:48,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:49,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:18:49,155 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:18:49,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:18:50,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:50,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:50,613 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:18:52,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:52,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:52,581 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:53,615 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:54,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:54,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:56,582 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:18:57,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:57,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:58,686 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:18:59,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:18:59,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:02,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:02,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:02,584 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:03,804 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:04,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:19:04,156 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:19:04,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:19:04,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:04,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:08,227 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:19:08,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:08,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:08,904 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:10,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:11,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:11,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:13,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:13,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:13,905 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:15,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:15,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:16,589 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:17,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:17,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:19,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:19:19,156 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:19:19,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:19:19,314 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:19,316 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:19:20,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:20,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:20,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:22,591 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:22,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:22,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:24,591 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:24,826 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:25,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:25,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:27,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:27,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:28,593 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:30,326 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:30,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:30,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:32,594 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:32,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:32,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:34,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:19:34,156 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:19:34,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:19:34,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:34,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:35,816 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:36,596 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:36,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:36,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:40,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:40,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:40,816 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:42,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:42,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:44,599 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:45,806 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:19:45,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:45,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:45,930 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:48,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:48,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:49,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:19:49,156 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:19:49,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:19:49,375 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:19:50,601 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:50,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:50,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:51,377 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:53,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:53,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:55,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:55,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:56,603 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:19:57,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:19:57,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:57,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:19:59,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:00,325 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:20:00,326 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:20:00,327 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:20:00,327 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:20:00,604 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:20:00,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:01,605 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:02,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:02,605 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:02,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:02,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:04,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:20:04,156 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:20:04,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:20:04,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:04,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:06,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:06,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:07,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:08,607 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:10,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:10,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:12,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:12,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:12,362 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:14,610 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:15,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:15,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:17,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:17,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:17,483 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:19,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:20:19,156 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:20:19,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:20:20,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:20,368 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:20:20,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:20,612 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:23,370 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:24,239 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:20:24,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:24,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:26,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:26,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:26,613 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:28,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:28,672 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:29,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:31,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:31,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:32,615 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:33,792 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:34,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:20:34,156 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:20:34,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:20:34,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:34,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:36,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:36,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:38,617 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:38,896 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:39,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:39,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:41,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:41,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:43,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:43,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:43,998 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:44,619 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:45,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:45,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:48,999 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:49,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:49,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:49,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:20:49,156 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:20:49,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:20:50,341 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:20:50,620 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:51,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:51,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:53,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:53,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:54,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:20:56,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:56,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:56,622 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:20:58,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:20:58,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:00,213 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:01,754 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:21:01,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:01,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:02,625 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:04,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:21:04,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:21:04,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:21:04,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:04,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:05,355 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:06,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:06,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:08,627 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:08,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:08,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:10,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:10,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:10,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:12,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:12,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:14,629 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:15,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:16,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:16,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:18,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:18,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:19,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:21:19,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:21:19,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:21:19,344 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:21:20,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:20,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:20,631 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:20,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:23,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:23,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:25,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:25,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:25,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:26,633 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:28,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:28,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:30,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:30,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:30,780 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:32,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:32,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:32,635 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:34,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:21:34,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:21:34,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:21:35,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:35,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:36,383 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:38,636 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:39,081 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:21:39,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:39,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:41,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:41,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:42,006 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:43,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:43,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:45,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:45,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:46,639 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:47,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:47,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:47,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:49,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:21:49,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:21:49,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:21:49,384 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:21:51,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:51,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:52,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:52,641 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:21:53,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:53,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:56,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:56,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:57,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:21:58,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:58,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:21:58,643 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:00,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:00,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:02,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:02,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:02,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:04,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:22:04,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:22:04,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:22:04,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:04,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:04,645 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:07,364 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:07,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:07,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:09,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:09,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:10,647 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:12,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:12,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:12,573 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:16,146 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:22:16,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:16,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:16,649 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:17,688 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:18,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:18,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:19,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:22:19,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:22:19,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:22:19,367 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:22:21,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:21,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:22,651 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:22,791 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:23,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:23,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:25,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:25,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:27,902 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:28,653 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:28,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:28,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:30,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:30,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:32,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:32,904 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:33,086 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:22:33,088 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:22:33,088 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:22:33,088 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:22:33,654 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:22:33,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:34,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:22:34,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:22:34,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:22:34,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:35,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:35,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:35,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:38,317 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:38,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:38,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:40,657 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:40,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:40,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:44,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:44,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:44,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:46,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:46,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:46,659 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:48,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:48,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:49,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:22:49,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:22:49,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:22:49,280 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:49,328 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:22:50,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:50,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:52,660 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:53,909 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:22:53,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:53,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:54,776 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:22:56,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:56,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:58,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:22:58,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:58,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:22:59,879 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:01,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:01,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:03,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:03,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:04,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:23:04,157 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:23:04,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:23:04,665 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:05,296 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:06,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:07,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:09,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:09,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:10,666 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:11,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:11,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:11,116 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:13,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:13,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:16,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:16,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:16,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:18,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:18,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:18,669 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:19,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:23:19,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:23:19,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:23:19,329 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:23:21,331 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:21,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:21,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:23,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:23,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:24,671 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:26,350 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:26,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:26,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:28,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:28,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:30,673 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:31,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:32,263 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:23:32,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:32,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:34,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:23:34,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:23:34,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:23:35,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:35,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:36,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:36,675 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:37,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:37,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:39,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:39,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:41,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:41,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:41,690 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:42,677 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:44,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:44,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:46,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:46,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:46,691 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:48,680 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:49,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:23:49,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:23:49,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:23:49,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:49,332 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:23:49,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:51,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:51,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:52,333 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:54,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:54,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:54,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:23:56,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:56,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:57,900 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:23:58,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:23:58,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:00,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:00,684 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:00,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:03,032 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:03,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:03,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:04,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:24:04,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:24:04,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:24:05,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:05,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:06,686 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:08,496 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:09,670 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:24:09,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:09,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:12,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:12,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:12,688 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:13,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:14,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:14,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:17,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:17,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:18,690 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:18,715 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:19,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:24:19,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:24:19,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:24:19,341 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:24:19,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:19,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:21,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:21,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:23,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:23,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:23,829 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:24,693 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:26,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:26,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:28,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:28,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:28,829 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:30,694 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:30,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:30,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:33,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:33,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:33,943 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:34,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:24:34,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:24:34,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:24:35,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:35,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:36,696 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:38,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:38,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:39,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:40,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:40,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:42,698 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:44,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:44,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:44,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:47,815 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:24:47,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:47,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:49,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:24:49,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:24:49,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:24:49,357 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:24:49,357 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:49,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:49,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:50,701 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:53,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:53,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:54,358 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:24:55,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:55,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:56,703 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:24:58,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:58,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:24:59,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:00,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:00,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:02,705 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:03,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:03,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:04,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:25:04,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:25:04,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:25:05,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:05,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:05,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:06,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:25:06,687 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:25:06,687 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:25:06,687 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:25:06,706 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:25:07,707 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:08,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:08,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:08,707 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:10,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:10,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:10,688 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:12,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:12,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:14,709 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:15,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:15,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:15,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:17,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:17,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:19,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:25:19,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:25:19,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:25:19,397 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:25:19,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:19,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:20,711 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:21,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:22,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:22,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:26,233 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:25:26,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:26,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:26,712 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:26,971 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:29,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:29,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:31,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:31,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:32,089 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:32,715 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:33,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:33,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:34,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:25:34,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:25:34,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:25:35,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:35,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:37,206 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:37,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:37,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:38,717 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:39,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:40,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:42,207 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:42,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:42,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:44,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:44,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:44,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:46,720 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:47,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:47,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:47,813 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:48,720 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:49,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:25:49,158 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:25:49,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:25:49,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:49,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:50,811 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:25:52,722 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:52,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:52,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:53,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:54,722 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:54,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:55,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:57,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:57,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:58,723 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:25:58,904 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:25:59,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:25:59,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:02,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:02,870 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:26:02,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:02,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:04,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:26:04,159 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:26:04,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:26:04,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:04,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:04,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:04,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:06,726 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:08,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:08,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:10,100 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:10,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:10,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:10,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:13,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:13,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:15,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:15,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:15,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:16,730 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:17,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:17,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:19,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:26:19,159 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:26:19,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:26:19,340 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:26:19,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:19,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:20,341 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:22,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:22,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:22,731 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:25,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:25,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:25,872 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:27,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:27,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:28,733 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:29,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:29,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:31,847 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:32,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:32,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:34,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:26:34,159 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:26:34,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:26:34,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:34,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:34,735 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:37,349 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:37,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:37,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:38,737 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:40,971 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:26:40,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:41,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:42,648 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:43,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:43,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:44,738 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:46,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:46,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:48,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:48,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:48,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:49,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:26:49,159 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:26:49,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:26:49,359 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:26:50,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:50,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:50,741 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:53,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:53,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:54,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:26:54,742 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:26:55,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:55,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:57,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:57,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:26:59,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:00,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:00,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:00,744 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:02,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:02,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:04,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:27:04,159 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:27:04,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:27:04,327 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:05,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:27:05,185 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:27:05,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:27:05,186 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:27:05,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:05,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:05,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:27:05,746 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:06,746 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:07,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:07,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:09,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:09,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:10,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:12,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:12,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:12,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:14,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:14,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:16,053 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:17,751 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:18,389 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:27:18,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:18,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:19,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:27:19,160 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:27:19,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:27:19,344 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:27:21,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:21,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:21,345 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:23,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:23,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:23,753 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:25,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:25,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:26,611 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:28,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:28,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:29,755 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:30,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:30,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:32,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:32,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:32,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:34,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:27:34,159 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:27:34,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:27:35,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:35,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:35,756 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:37,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:37,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:38,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:39,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:39,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:39,758 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:42,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:42,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:43,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:44,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:44,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:45,760 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:47,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:47,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:49,072 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:49,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:27:49,159 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:27:49,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:27:49,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:49,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:49,379 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:27:51,762 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:52,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:52,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:54,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:27:55,988 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:27:56,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:56,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:57,764 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:27:58,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:58,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:27:59,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:01,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:01,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:01,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:03,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:03,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:04,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:28:04,160 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:28:04,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:28:05,291 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:05,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:05,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:07,767 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:08,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:08,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:10,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:10,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:10,848 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:13,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:13,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:13,769 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:15,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:15,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:16,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:17,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:18,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:18,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:19,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:28:19,160 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:28:19,180 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:28:19,377 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:28:20,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:20,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:21,378 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:22,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:22,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:25,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:25,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:26,379 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:27,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:27,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:29,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:29,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:31,380 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:32,971 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:28:32,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:33,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:34,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:28:34,160 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:28:34,180 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:28:35,776 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:35,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:35,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:37,363 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:37,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:37,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:40,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:40,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:41,778 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:42,643 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:42,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:42,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:44,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:44,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:45,779 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:48,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:48,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:48,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:49,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:28:49,160 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:28:49,180 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:28:50,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:50,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:50,524 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:28:51,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:52,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:52,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:53,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:28:55,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:55,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:57,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:57,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:28:57,782 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:28:58,979 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:00,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:00,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:02,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:02,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:03,784 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:04,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:29:04,160 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:29:04,180 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:29:04,364 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:04,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:04,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:07,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:07,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:07,785 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:09,372 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:10,753 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:29:10,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:10,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:12,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:12,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:13,787 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:14,733 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:16,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:16,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:18,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:18,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:19,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:29:19,160 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:29:19,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:29:19,789 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:20,447 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:29:20,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:21,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:21,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:23,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:23,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:23,791 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:26,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:26,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:26,290 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:28,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:28,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:29,794 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:30,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:30,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:32,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:33,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:33,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:34,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:29:34,161 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:29:34,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:29:34,218 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:29:34,320 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:29:34,320 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:29:34,320 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:29:34,795 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:29:35,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:35,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:35,796 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:37,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:37,321 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:37,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:40,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:40,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:41,798 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:42,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:42,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:43,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:45,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:45,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:47,800 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:49,104 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:49,125 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:29:49,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:49,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:49,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:29:49,161 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:29:49,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:29:49,358 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:29:51,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:52,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:52,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:54,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:54,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:54,984 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:29:56,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:56,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:57,803 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:29:59,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:29:59,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:00,944 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:01,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:01,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:03,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:03,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:03,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:04,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:30:04,161 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:30:04,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:30:06,288 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:06,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:06,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:08,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:08,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:09,807 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:11,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:11,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:11,742 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:13,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:13,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:15,809 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:16,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:16,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:17,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:18,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:18,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:19,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:30:19,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:30:19,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:30:19,353 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:30:19,810 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:20,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:20,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:23,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:23,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:23,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:25,813 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:27,160 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:30:27,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:27,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:28,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:29,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:29,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:31,815 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:32,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:32,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:34,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:34,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:34,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:34,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:30:34,161 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:30:34,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:30:35,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:37,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:37,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:39,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:39,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:39,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:41,818 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:42,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:42,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:44,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:44,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:45,057 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:46,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:46,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:47,820 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:48,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:48,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:49,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:30:49,161 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:30:49,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:30:50,817 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:30:50,818 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:51,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:51,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:53,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:53,822 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:30:53,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:55,916 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:30:56,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:56,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:58,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:58,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:30:59,824 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:01,711 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:01,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:01,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:03,825 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:04,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:31:04,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:31:04,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:31:05,554 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:31:05,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:05,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:07,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:07,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:07,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:09,827 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:10,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:10,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:12,643 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:12,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:12,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:14,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:14,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:15,829 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:16,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:16,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:18,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:19,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:31:19,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:31:19,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:31:19,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:19,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:19,831 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:20,385 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:31:21,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:21,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:23,386 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:24,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:24,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:25,833 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:26,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:26,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:28,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:29,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:29,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:31,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:31,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:31,834 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:33,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:33,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:34,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:31:34,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:31:34,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:31:34,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:35,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:35,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:37,836 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:39,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:39,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:39,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:42,686 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:31:42,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:42,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:43,838 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:44,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:44,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:45,074 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:47,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:47,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:47,840 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:49,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:31:49,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:31:49,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:31:49,357 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:31:49,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:49,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:50,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:52,215 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:31:52,216 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:31:52,216 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:31:52,217 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:31:52,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:31:52,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:52,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:53,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:31:54,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:54,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:56,218 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:31:56,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:56,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:59,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:59,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:31:59,844 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:01,868 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:02,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:02,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:04,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:32:04,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:32:04,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:32:04,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:04,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:05,846 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:06,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:07,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:07,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:09,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:09,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:11,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:11,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:11,849 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:12,778 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:14,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:14,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:15,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:16,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:16,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:18,600 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:19,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:32:19,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:32:19,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:32:19,377 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:32:20,124 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:32:20,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:20,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:21,852 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:22,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:22,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:24,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:25,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:25,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:25,853 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:27,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:27,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:29,650 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:30,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:30,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:31,856 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:32,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:32,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:34,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:32:34,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:32:34,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:32:35,386 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:35,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:35,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:37,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:37,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:37,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:40,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:40,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:40,993 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:42,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:42,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:43,860 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:44,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:44,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:45,993 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:47,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:48,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:49,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:32:49,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:32:49,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:32:49,361 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:32:49,862 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:50,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:50,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:51,362 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:52,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:52,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:53,863 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:32:55,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:55,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:56,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:32:58,793 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:32:58,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:58,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:32:59,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:01,734 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:01,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:01,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:03,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:03,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:04,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:33:04,163 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:33:04,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:33:05,866 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:06,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:06,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:07,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:08,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:08,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:09,869 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:10,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:10,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:12,483 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:13,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:13,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:15,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:15,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:15,870 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:18,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:18,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:18,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:19,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:33:19,162 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:33:19,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:33:19,387 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:33:20,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:20,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:21,872 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:23,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:23,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:24,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:25,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:25,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:27,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:28,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:28,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:29,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:30,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:30,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:33,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:33,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:33,877 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:34,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:33:34,163 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:33:34,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:33:34,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:36,644 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:33:36,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:36,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:37,878 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:38,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:38,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:40,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:41,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:41,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:43,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:43,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:43,881 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:45,978 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:46,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:46,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:48,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:48,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:49,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:33:49,163 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:33:49,183 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:33:49,364 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:33:49,883 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:51,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:51,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:51,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:53,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:53,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:53,885 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:33:56,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:56,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:56,726 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:33:58,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:58,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:33:59,886 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:01,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:01,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:02,642 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:03,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:03,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:04,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:34:04,163 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:34:04,183 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:34:05,889 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:05,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:06,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:08,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:08,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:08,335 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:09,386 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:34:09,388 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:34:09,388 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:34:09,389 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:34:09,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:34:09,891 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:10,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:10,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:11,891 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:13,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:14,714 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:34:14,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:14,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:17,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:17,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:17,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:19,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:34:19,163 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:34:19,183 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:34:19,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:19,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:20,963 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:34:20,964 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:22,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:22,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:23,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:24,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:24,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:25,965 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:27,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:27,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:27,896 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:29,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:29,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:31,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:32,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:32,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:33,898 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:34,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:34:34,163 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:34:34,183 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:34:34,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:34,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:36,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:37,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:37,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:37,899 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:39,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:39,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:41,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:41,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:41,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:43,901 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:44,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:44,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:46,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:46,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:47,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:49,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:34:49,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:34:49,183 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:34:49,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:49,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:49,903 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:50,835 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:34:52,836 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:52,927 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:34:52,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:52,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:55,905 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:34:55,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:56,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:58,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:58,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:34:58,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:34:59,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:00,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:00,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:02,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:02,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:03,734 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:04,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:35:04,163 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:35:04,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:35:05,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:05,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:05,909 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:07,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:07,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:09,370 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:10,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:10,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:11,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:12,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:12,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:14,984 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:15,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:15,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:15,914 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:17,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:17,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:19,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:35:19,163 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:35:19,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:35:20,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:20,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:20,942 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:35:20,943 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:21,916 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:22,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:22,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:25,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:25,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:25,945 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:27,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:27,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:27,917 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:30,750 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:35:30,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:30,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:31,318 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:33,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:33,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:33,919 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:34,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:35:34,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:35:34,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:35:35,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:35,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:36,321 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:38,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:38,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:39,921 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:40,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:40,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:41,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:42,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:42,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:43,922 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:45,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:45,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:47,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:47,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:47,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:49,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:35:49,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:35:49,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:35:49,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:50,432 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:35:50,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:50,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:52,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:52,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:52,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:55,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:55,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:55,927 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:35:57,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:57,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:35:57,845 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:35:59,929 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:00,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:00,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:02,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:02,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:03,681 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:04,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:36:04,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:36:04,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:36:05,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:05,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:05,931 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:08,841 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:36:08,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:08,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:09,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:11,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:11,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:11,933 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:13,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:13,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:14,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:15,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:15,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:17,936 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:18,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:18,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:19,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:36:19,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:36:19,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:36:19,372 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:36:20,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:20,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:20,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:22,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:22,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:23,943 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:25,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:25,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:25,910 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:25,981 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:36:25,983 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:36:25,983 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:36:25,983 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:36:26,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:36:27,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:27,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:27,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:30,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:30,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:30,984 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:32,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:32,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:33,949 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:34,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:36:34,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:36:34,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:36:35,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:35,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:36,380 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:37,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:37,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:37,950 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:40,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:40,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:41,996 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:42,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:42,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:43,953 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:46,224 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:36:46,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:46,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:47,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:49,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:49,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:49,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:36:49,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:36:49,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:36:49,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:50,292 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:36:51,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:51,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:53,000 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:53,957 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:36:53,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:53,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:56,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:56,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:58,001 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:36:58,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:58,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:36:59,959 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:00,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:00,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:02,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:02,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:03,849 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:04,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:37:04,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:37:04,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:37:04,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:04,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:05,961 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:07,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:07,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:09,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:09,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:09,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:11,963 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:12,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:12,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:14,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:14,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:14,626 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:17,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:17,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:17,964 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:19,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:37:19,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:37:19,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:37:19,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:19,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:19,375 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:37:20,201 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:21,966 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:22,708 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:37:22,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:22,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:25,202 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:25,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:25,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:27,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:27,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:27,967 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:30,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:30,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:31,046 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:32,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:32,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:33,969 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:34,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:37:34,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:37:34,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:37:35,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:35,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:36,291 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:37,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:37,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:37,971 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:40,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:40,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:41,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:42,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:42,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:43,972 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:45,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:45,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:47,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:47,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:47,611 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:49,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:37:49,165 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:37:49,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:37:49,377 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:37:49,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:49,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:49,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:51,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:52,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:53,378 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:37:55,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:55,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:55,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:37:57,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:57,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:37:58,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:00,395 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:38:00,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:00,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:01,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:03,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:03,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:04,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:38:04,165 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:38:04,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:38:04,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:05,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:05,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:05,981 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:08,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:08,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:09,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:10,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:10,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:11,982 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:13,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:13,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:15,073 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:15,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:15,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:17,984 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:18,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:18,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:19,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:38:19,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:38:19,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:38:19,379 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:38:20,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:20,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:20,380 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:21,986 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:23,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:23,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:25,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:25,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:25,671 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:27,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:27,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:27,988 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:29,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:29,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:31,268 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:32,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:32,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:33,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:34,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:38:34,165 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:38:34,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:38:34,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:34,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:36,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:37,992 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:38,105 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:38:38,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:38,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:40,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:40,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:42,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:42,868 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:38:42,869 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:38:42,870 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:38:42,870 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:38:42,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:42,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:42,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:38:43,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:44,995 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:45,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:45,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:47,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:47,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:47,873 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:49,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:38:49,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:38:49,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:38:49,394 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:38:50,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:50,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:52,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:52,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:53,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:38:55,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:55,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:55,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:38:57,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:57,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:38:59,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:00,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:00,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:02,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:02,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:04,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:39:04,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:39:04,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:39:05,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:05,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:05,343 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:07,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:07,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:09,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:09,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:10,003 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:39:10,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:11,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:11,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:15,285 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:39:15,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:15,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:15,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:17,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:17,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:19,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:39:19,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:39:19,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:39:19,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:19,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:20,908 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:39:20,908 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:22,008 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:39:22,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:22,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:24,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:24,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:25,910 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:27,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:27,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:29,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:29,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:30,910 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:32,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:32,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:34,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:39:34,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:39:34,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:39:34,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:34,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:36,013 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:39:36,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:37,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:37,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:39,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:39,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:41,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:42,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:42,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:44,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:44,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:46,721 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:46,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:46,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:48,017 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:39:48,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:48,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:49,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:39:49,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:39:49,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:39:49,384 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:39:52,385 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:39:52,439 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:39:52,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:52,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:55,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:55,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:57,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:57,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:39:57,386 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:00,021 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:40:00,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:00,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:02,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:02,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:03,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:04,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:40:04,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:40:04,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:40:05,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:05,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:07,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:07,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:08,325 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:10,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:10,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:12,025 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:40:12,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:12,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:14,294 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:14,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:14,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:17,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:17,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:19,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:40:19,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:40:19,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:40:19,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:19,386 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:40:19,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:19,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:22,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:22,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:24,029 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:40:24,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:24,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:24,757 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:26,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:26,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:29,758 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:29,803 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:40:29,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:29,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:32,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:32,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:34,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:40:34,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:40:34,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:40:34,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:34,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:35,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:37,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:37,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:38,033 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:40:39,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:39,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:40,817 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:42,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:42,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:44,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:44,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:45,818 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:47,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:47,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:49,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:40:49,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:40:49,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:40:49,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:49,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:50,037 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:40:51,134 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:40:51,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:52,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:52,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:54,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:54,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:56,136 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:40:57,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:57,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:59,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:40:59,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:01,761 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:01,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:01,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:02,041 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:41:03,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:03,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:04,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:41:04,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:41:04,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:41:07,277 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:41:07,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:07,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:07,376 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:10,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:10,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:12,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:12,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:12,377 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:14,045 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:41:14,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:14,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:16,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:16,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:18,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:19,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:41:19,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:41:19,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:41:19,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:19,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:20,174 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:41:21,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:21,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:23,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:24,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:24,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:26,049 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:41:26,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:26,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:28,222 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:29,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:29,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:31,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:31,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:33,223 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:34,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:41:34,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:41:34,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:41:34,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:34,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:36,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:36,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:38,053 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:41:38,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:39,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:39,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:41,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:41,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:43,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:44,626 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:41:44,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:44,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:47,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:47,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:48,355 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:49,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:41:49,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:41:49,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:41:49,391 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:41:49,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:49,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:52,058 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:41:52,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:52,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:53,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:54,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:54,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:57,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:57,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:58,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:41:59,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:41:59,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:02,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:02,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:04,063 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:42:04,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:42:04,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:42:04,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:04,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:42:04,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:04,358 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:07,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:07,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:09,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:09,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:09,358 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:11,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:11,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:13,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:13,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:14,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:16,066 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:42:16,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:16,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:18,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:18,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:19,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:42:19,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:42:19,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:42:19,392 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:42:19,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:22,098 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:42:22,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:22,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:24,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:24,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:24,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:26,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:26,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:28,069 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:42:29,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:29,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:29,954 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:31,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:31,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:34,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:42:34,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:42:34,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:42:34,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:34,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:35,344 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:36,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:36,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:39,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:39,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:40,072 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:42:41,308 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:41,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:41,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:44,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:44,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:46,309 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:46,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:46,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:49,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:49,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:49,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:42:49,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:42:49,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:42:50,578 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:42:51,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:51,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:51,579 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:54,076 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:42:54,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:54,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:56,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:56,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:56,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:42:59,480 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:42:59,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:42:59,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:01,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:02,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:02,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:04,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:43:04,168 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:43:04,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:43:04,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:04,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:06,080 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:43:07,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:07,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:07,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:09,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:09,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:12,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:12,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:12,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:14,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:14,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:17,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:17,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:18,083 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:43:18,094 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:19,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:19,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:19,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:43:19,168 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:43:19,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:43:19,396 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:43:21,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:21,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:23,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:23,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:24,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:26,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:26,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:28,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:28,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:28,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:31,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:31,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:32,087 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:43:33,629 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:33,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:33,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:34,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:43:34,167 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:43:34,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:43:37,360 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:43:37,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:37,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:39,321 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:40,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:40,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:42,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:42,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:44,091 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:43:44,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:44,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:44,881 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:46,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:46,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:49,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:43:49,168 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:43:49,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:43:49,397 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:43:49,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:49,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:50,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:51,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:51,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:54,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:54,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:56,095 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:43:56,210 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:43:56,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:56,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:59,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:43:59,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:01,210 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:01,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:01,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:04,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:44:04,168 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:44:04,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:44:04,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:04,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:06,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:06,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:06,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:08,099 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:44:09,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:09,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:11,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:11,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:11,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:14,687 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:44:14,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:14,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:16,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:17,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:17,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:19,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:44:19,168 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:44:19,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:44:19,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:19,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:20,103 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:44:21,409 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:44:21,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:22,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:22,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:24,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:24,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:26,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:27,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:27,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:29,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:29,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:32,077 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:32,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:32,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:33,107 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:44:34,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:44:34,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:44:34,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:44:34,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:34,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:37,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:37,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:37,337 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:39,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:39,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:42,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:42,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:42,338 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:44,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:44,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:45,110 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:44:47,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:47,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:47,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:49,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:49,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:49,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:44:49,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:44:49,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:44:49,402 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:44:52,353 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:44:52,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:52,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:53,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:55,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:55,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:57,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:57,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:58,738 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:44:59,115 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:44:59,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:44:59,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:01,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:01,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:03,739 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:04,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:45:04,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:45:04,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:45:04,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:04,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:06,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:06,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:09,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:09,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:09,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:11,119 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:45:11,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:11,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:14,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:14,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:14,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:16,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:16,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:19,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:45:19,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:45:19,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:45:19,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:19,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:19,306 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:19,403 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:45:21,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:21,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:23,123 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:45:23,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:23,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:24,490 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:26,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:26,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:29,388 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:45:29,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:29,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:29,490 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:32,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:32,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:34,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:34,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:34,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:45:34,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:45:34,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:45:35,127 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:45:35,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:37,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:37,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:39,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:39,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:40,356 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:41,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:41,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:43,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:43,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:45,790 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:45,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:45,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:47,131 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:45:48,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:48,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:49,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:45:49,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:45:49,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:45:49,404 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:45:50,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:50,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:51,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:53,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:53,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:55,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:55,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:56,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:45:58,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:58,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:45:59,136 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:46:00,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:00,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:01,702 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:03,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:03,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:04,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:46:04,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:46:04,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:46:06,739 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:46:06,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:06,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:07,363 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:09,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:09,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:11,139 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:46:11,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:11,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:12,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:14,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:14,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:16,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:16,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:17,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:19,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:46:19,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:46:19,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:46:19,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:19,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:21,270 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:46:21,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:21,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:22,604 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:23,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:46:24,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:24,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:26,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:26,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:27,604 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:29,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:29,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:31,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:31,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:32,605 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:34,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:46:34,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:46:34,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:46:34,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:34,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:36,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:36,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:37,147 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:46:38,311 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:39,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:39,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:41,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:41,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:43,312 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:44,567 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:46:44,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:44,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:47,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:47,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:48,596 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:49,150 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:46:49,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:46:49,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:46:49,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:46:49,407 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:46:49,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:49,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:52,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:52,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:54,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:54,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:54,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:46:57,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:57,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:59,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:46:59,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:00,093 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:01,154 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:47:01,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:01,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:04,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:04,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:04,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:47:04,169 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:47:04,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:47:05,324 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:06,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:06,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:08,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:08,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:10,325 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:11,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:11,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:13,158 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:47:13,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:13,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:15,904 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:16,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:16,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:18,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:18,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:19,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:47:19,170 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:47:19,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:47:19,409 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:47:21,081 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:21,842 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:47:21,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:21,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:23,161 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:47:24,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:24,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:26,688 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:27,162 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:47:27,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:27,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:29,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:30,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:32,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:32,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:33,164 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:47:34,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:34,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:47:34,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:47:34,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:47:37,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:37,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:37,867 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:39,166 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:47:39,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:39,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:42,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:42,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:42,868 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:44,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:44,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:47,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:47,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:47,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:49,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:47:49,170 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:47:49,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:47:49,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:49,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:49,410 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:47:51,170 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:47:52,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:52,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:53,790 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:54,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:54,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:56,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:56,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:58,790 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:47:58,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:47:59,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:01,174 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:48:02,294 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:48:02,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:02,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:04,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:48:04,170 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:48:04,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:48:04,375 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:05,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:05,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:07,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:07,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:09,375 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:10,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:10,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:12,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:12,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:13,178 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:48:14,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:14,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:15,035 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:16,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:16,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:19,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:48:19,170 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:48:19,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:48:19,411 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:48:19,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:19,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:20,964 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:21,181 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:48:21,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:21,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:24,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:24,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:25,965 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:26,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:26,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:29,183 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:48:29,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:29,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:31,684 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:31,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:31,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:34,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:48:34,170 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:48:34,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:48:34,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:34,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:36,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:36,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:37,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:39,910 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:48:39,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:39,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:42,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:42,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:43,187 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:48:43,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:44,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:44,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:47,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:47,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:48,374 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:49,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:48:49,170 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:48:49,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:48:49,413 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:48:49,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:49,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:52,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:52,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:53,191 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:48:54,078 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:54,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:54,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:57,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:57,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:59,080 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:48:59,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:48:59,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:02,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:02,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:04,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:49:04,171 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:49:04,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:49:04,294 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:04,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:04,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:05,195 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:49:07,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:07,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:09,294 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:09,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:09,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:12,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:12,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:14,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:14,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:14,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:15,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:49:17,474 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:49:17,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:17,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:19,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:49:19,171 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:49:19,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:49:20,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:20,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:20,540 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:49:20,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:22,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:22,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:25,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:25,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:25,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:27,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:27,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:29,203 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:49:30,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:30,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:30,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:32,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:32,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:34,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:49:34,171 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:49:34,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:49:34,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:34,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:36,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:36,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:36,959 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:37,206 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:49:39,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:39,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:41,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:41,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:41,960 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:44,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:44,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:45,209 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:49:46,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:46,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:47,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:49,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:49:49,171 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:49:49,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:49:49,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:49,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:49,416 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:49:51,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:51,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:52,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:54,705 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:49:54,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:54,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:57,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:57,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:58,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:49:59,214 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:49:59,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:49:59,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:02,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:02,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:03,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:04,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:50:04,171 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:50:04,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:50:04,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:04,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:07,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:07,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:08,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:09,217 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:50:09,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:09,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:12,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:12,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:13,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:14,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:14,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:16,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:16,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:18,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:18,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:18,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:19,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:50:19,171 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:50:19,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:50:19,417 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:50:21,221 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:50:21,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:21,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:23,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:23,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:24,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:26,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:26,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:28,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:28,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:29,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:31,225 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:50:32,269 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:50:32,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:32,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:33,866 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:34,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:50:34,171 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:50:34,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:50:35,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:35,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:37,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:37,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:39,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:39,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:39,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:41,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:41,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:43,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:50:44,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:44,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:45,113 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:46,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:46,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:49,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:50:49,171 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:50:49,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:50:49,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:49,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:50,665 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:50:50,667 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:51,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:51,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:54,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:54,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:55,666 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:50:56,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:56,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:57,234 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:50:59,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:50:59,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:00,758 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:01,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:01,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:04,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:51:04,172 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:51:04,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:04,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:51:04,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:05,238 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:51:06,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:06,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:06,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:09,493 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:51:09,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:09,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:11,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:12,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:12,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:14,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:14,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:16,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:17,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:17,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:19,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:19,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:19,171 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:51:19,172 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:51:19,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:51:19,242 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:51:20,638 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:51:21,639 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:22,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:22,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:24,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:24,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:26,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:26,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:27,251 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:51:27,599 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:28,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:29,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:31,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:31,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:32,599 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:33,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:33,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:34,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:51:34,172 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:51:34,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:51:36,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:36,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:38,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:38,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:38,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:41,257 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:51:41,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:41,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:43,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:43,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:43,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:46,943 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:51:46,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:46,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:48,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:49,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:51:49,172 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:51:49,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:51:49,260 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:51:49,423 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:51:49,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:49,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:51,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:51,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:54,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:54,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:54,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:56,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:56,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:59,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:51:59,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:51:59,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:01,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:01,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:03,265 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:52:04,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:52:04,172 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:52:04,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:52:04,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:04,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:05,335 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:06,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:06,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:09,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:09,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:10,336 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:11,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:11,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:11,268 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:52:14,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:14,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:15,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:16,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:16,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:18,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:18,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:19,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:52:19,172 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:52:19,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:52:19,424 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:52:20,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:20,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:21,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:24,225 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:52:24,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:24,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:25,277 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:52:27,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:27,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:27,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:29,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:29,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:31,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:31,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:32,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:33,280 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:52:33,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:33,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:34,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:52:34,173 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:52:34,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:52:36,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:36,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:38,347 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:38,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:38,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:41,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:41,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:43,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:43,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:43,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:46,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:46,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:47,286 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:52:48,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:48,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:48,958 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:49,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:52:49,173 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:52:49,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:52:50,828 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:52:51,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:51,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:53,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:53,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:54,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:52:55,288 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:52:56,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:56,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:58,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:58,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:52:59,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:01,579 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:53:01,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:01,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:04,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:53:04,173 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:53:04,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:53:04,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:04,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:05,293 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:53:05,333 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:06,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:06,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:09,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:09,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:10,334 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:11,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:11,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:14,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:14,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:15,296 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:53:15,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:16,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:16,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:18,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:18,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:19,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:53:19,173 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:53:19,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:53:19,428 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:53:21,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:21,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:21,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:23,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:23,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:25,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:25,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:26,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:28,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:28,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:29,302 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:53:30,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:30,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:31,644 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:33,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:33,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:34,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:53:34,173 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:53:34,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:53:35,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:35,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:37,384 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:38,822 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:53:38,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:38,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:39,306 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:53:41,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:41,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:42,734 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:43,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:43,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:46,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:46,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:47,735 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:48,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:48,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:49,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:53:49,173 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:53:49,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:53:49,430 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:53:51,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:51,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:53,004 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:53,311 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:53:53,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:53,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:56,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:56,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:58,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:53:58,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:53:58,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:01,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:01,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:03,006 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:03,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:03,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:04,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:54:04,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:54:04,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:54:05,317 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:54:06,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:06,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:08,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:08,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:08,628 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:11,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:11,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:13,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:13,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:13,629 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:16,410 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:54:16,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:16,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:17,321 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:54:19,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:19,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:19,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:54:19,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:54:19,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:54:19,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:19,433 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:54:21,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:21,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:23,322 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:54:23,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:23,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:25,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:25,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:25,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:28,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:28,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:30,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:30,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:31,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:31,330 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:54:33,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:33,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:34,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:54:34,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:54:34,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:54:35,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:35,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:36,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:38,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:38,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:40,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:40,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:41,321 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:43,334 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:54:43,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:43,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:45,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:45,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:46,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:48,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:48,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:49,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:54:49,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:54:49,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:54:49,433 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:54:50,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:50,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:51,979 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:53,337 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:54:53,689 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:54:53,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:53,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:56,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:56,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:56,986 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:54:58,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:54:58,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:01,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:01,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:01,987 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:03,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:03,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:04,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:55:04,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:55:04,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:55:06,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:06,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:07,341 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:55:07,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:08,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:08,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:11,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:11,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:12,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:13,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:13,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:16,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:16,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:17,823 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:18,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:18,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:19,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:55:19,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:55:19,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:55:19,344 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:55:19,435 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:55:20,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:20,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:22,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:22,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:23,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:25,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:25,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:27,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:27,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:29,224 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:29,347 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:55:31,041 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:55:31,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:31,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:33,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:33,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:34,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:55:34,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:55:34,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:55:34,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:35,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:35,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:38,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:38,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:39,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:40,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:40,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:41,351 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:55:43,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:43,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:44,561 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:45,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:45,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:48,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:48,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:49,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:55:49,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:55:49,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:55:49,437 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:55:50,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:50,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:50,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:51,354 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:55:53,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:53,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:55,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:55,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:55,694 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:55:58,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:55:58,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:00,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:00,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:00,695 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:03,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:03,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:04,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:56:04,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:56:04,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:56:05,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:05,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:05,359 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:56:06,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:08,771 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:56:08,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:08,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:10,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:10,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:11,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:13,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:13,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:15,689 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:15,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:17,136 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:17,362 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:56:17,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:17,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:19,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:56:19,174 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:56:19,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:56:20,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:20,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:20,881 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:56:22,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:22,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:22,882 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:25,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:25,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:27,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:27,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:27,884 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:29,365 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:56:30,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:30,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:32,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:32,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:33,796 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:34,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:56:34,175 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:56:34,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:56:35,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:35,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:37,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:37,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:38,966 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:40,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:40,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:40,368 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:56:42,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:42,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:44,010 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:45,392 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:56:45,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:45,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:48,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:48,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:49,011 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:49,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:56:49,175 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:56:49,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:56:49,440 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:56:50,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:50,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:53,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:53,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:54,373 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:56:54,562 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:56:55,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:55,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:57,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:58,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:56:59,563 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:00,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:00,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:02,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:02,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:04,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:57:04,176 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:57:04,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:57:04,376 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:57:04,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:04,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:05,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:07,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:07,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:09,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:09,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:10,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:12,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:12,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:14,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:14,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:15,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:17,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:17,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:18,380 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:57:19,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:57:19,175 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:57:19,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:57:19,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:19,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:20,600 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:57:20,601 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:23,072 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:57:23,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:23,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:25,601 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:26,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:26,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:28,383 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:57:28,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:28,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:30,759 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:31,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:31,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:33,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:33,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:34,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:57:34,175 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:57:34,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:57:36,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:36,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:36,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 20:57:36,256 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 20:57:36,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:36,438 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 20:57:36,438 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 20:57:36,438 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 20:57:36,438 DEBUG SenderThread:1893870 [sender.py:send():382] send: metric 2024-03-27 20:57:36,438 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 20:57:36,439 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 20:57:36,439 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 20:57:37,385 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 20:57:37,386 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/config.yaml 2024-03-27 20:57:38,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:38,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:38,386 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:57:41,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:41,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:41,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:43,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:43,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:46,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:46,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:46,389 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:57:47,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:48,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:48,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:49,443 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:57:49,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:57:49,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:57:49,501 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:57:50,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:50,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:52,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:52,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:53,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:55,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:55,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:57,647 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:57:58,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:57:58,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:02,447 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:58:02,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:02,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:02,648 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:04,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:58:04,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:58:04,426 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:58:05,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:05,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:07,825 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:08,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:08,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:08,395 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/config.yaml 2024-03-27 20:58:08,395 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:10,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:10,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:12,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:12,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:13,284 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:13,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:15,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:15,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:16,397 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:18,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:18,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:18,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:19,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:58:19,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:58:19,426 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:58:19,573 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:58:20,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:20,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:22,399 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:23,367 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:23,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:23,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:25,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:25,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:28,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:28,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:28,401 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:28,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:30,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:30,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:33,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:33,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:33,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:34,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:34,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:58:34,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:58:34,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:58:35,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:35,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:38,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:38,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:38,556 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:40,405 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:41,976 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:58:41,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:42,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:44,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:45,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:45,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:46,406 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:47,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:47,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:49,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:58:49,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:58:49,426 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:58:49,551 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:58:49,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:50,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:50,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:52,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:52,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:52,409 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:58:55,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:55,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:55,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:58:57,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:57,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:58:58,413 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:00,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:00,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:00,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:02,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:02,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:04,415 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:04,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:59:04,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:59:04,426 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:59:05,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:05,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:05,633 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:07,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:07,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:10,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:10,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:10,416 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:11,604 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:12,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:12,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:15,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:15,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:16,418 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:16,640 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:17,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:17,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:19,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:59:19,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:59:19,426 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:59:20,576 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:59:20,834 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:59:20,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:20,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:21,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:22,420 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:23,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:23,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:25,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:25,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:26,669 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:28,422 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:28,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:28,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:30,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:30,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:31,691 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:33,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:33,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:34,423 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:34,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:59:34,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:59:34,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:59:35,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:35,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:37,636 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:38,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:38,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:40,425 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:40,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:40,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:42,754 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:44,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:44,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:46,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:46,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:46,427 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:47,778 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:49,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:49,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:49,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 20:59:49,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 20:59:49,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 20:59:49,587 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 20:59:51,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:51,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:52,430 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:52,832 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:54,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:54,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:56,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:56,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:57,832 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 20:59:58,432 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 20:59:59,900 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 20:59:59,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 20:59:59,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:02,849 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:02,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:02,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:04,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:00:04,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:00:04,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:00:04,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:04,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:06,434 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:07,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:07,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:08,610 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:09,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:09,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:12,436 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:13,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:13,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:13,906 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:15,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:15,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:18,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:18,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:18,438 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:18,927 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:19,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:00:19,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:00:19,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:00:19,618 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:00:20,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:20,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:22,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:22,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:23,966 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:24,439 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:25,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:25,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:27,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:27,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:28,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:29,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:29,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:30,441 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:32,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:32,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:33,992 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:34,090 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:00:34,092 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:00:34,092 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:00:34,093 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:00:34,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:00:34,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:00:34,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:00:34,442 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:00:34,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:34,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:36,443 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:38,402 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:00:38,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:38,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:39,658 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:40,444 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:41,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:41,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:43,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:43,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:44,446 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:44,807 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:45,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:45,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:46,446 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:48,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:48,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:49,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:00:49,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:00:49,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:00:49,627 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:00:50,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:50,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:50,447 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:50,628 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:52,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:53,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:53,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:55,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:55,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:56,009 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:00:57,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:57,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:58,449 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:00:59,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:00:59,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:01,418 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:01,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:01,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:04,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:01:04,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:01:04,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:01:04,451 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:04,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:04,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:06,647 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:06,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:06,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:09,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:09,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:10,453 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:11,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:11,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:11,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:15,362 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:01:15,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:15,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:16,455 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:17,012 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:18,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:18,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:19,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:01:19,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:01:19,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:01:20,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:20,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:20,573 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:01:22,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:22,574 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:23,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:23,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:25,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:25,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:27,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:27,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:28,090 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:28,458 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:30,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:30,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:32,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:32,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:33,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:34,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:34,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:34,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:01:34,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:01:34,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:01:34,460 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:36,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:37,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:38,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:39,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:39,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:40,462 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:41,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:42,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:43,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:44,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:44,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:46,464 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:47,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:47,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:48,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:49,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:49,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:49,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:01:49,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:01:49,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:01:50,563 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:01:52,466 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:52,839 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:01:52,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:52,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:53,564 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:01:55,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:55,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:57,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:57,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:01:58,468 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:01:59,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:00,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:00,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:02,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:02,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:04,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:04,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:02:04,427 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:02:04,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:02:04,470 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:05,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:05,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:07,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:07,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:09,322 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:09,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:09,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:10,472 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:12,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:12,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:14,322 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:14,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:14,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:16,474 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:17,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:17,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:19,382 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:19,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:02:19,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:02:19,428 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:02:19,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:19,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:20,910 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:02:21,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:21,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:22,477 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:24,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:24,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:24,912 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:26,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:26,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:28,479 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:30,059 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:02:30,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:30,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:30,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:32,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:32,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:34,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:02:34,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:02:34,428 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:02:34,481 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:34,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:34,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:35,575 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:37,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:38,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:40,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:40,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:40,483 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:41,498 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:42,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:42,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:45,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:45,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:46,485 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:46,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:47,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:48,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:49,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:02:49,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:02:49,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:02:50,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:50,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:50,406 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:02:50,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:02:50,654 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:02:50,654 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:02:50,655 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:02:51,486 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:02:51,655 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:52,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:52,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:52,487 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:02:54,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:54,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:57,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:57,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:02:57,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:02:58,490 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:00,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:00,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:02,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:02,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:02,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:04,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:03:04,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:03:04,428 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:03:04,492 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:04,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:04,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:08,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:08,584 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:03:08,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:08,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:10,494 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:10,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:10,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:13,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:13,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:13,660 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:15,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:15,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:16,496 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:18,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:18,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:18,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:19,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:03:19,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:03:19,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:03:19,563 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:03:20,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:20,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:22,498 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:23,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:23,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:23,717 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:25,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:25,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:28,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:28,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:28,499 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:28,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:30,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:30,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:33,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:33,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:33,778 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:34,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:03:34,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:03:34,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:03:34,501 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:35,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:35,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:38,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:38,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:38,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:40,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:40,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:40,503 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:43,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:43,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:43,812 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:46,505 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:47,090 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:03:47,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:47,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:48,838 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:49,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:03:49,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:03:49,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:03:49,655 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:03:50,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:50,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:52,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:52,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:52,507 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:53,878 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:03:55,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:55,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:57,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:57,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:03:58,509 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:03:58,905 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:00,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:00,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:02,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:02,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:03,924 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:04,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:04:04,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:04:04,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:04:04,511 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:05,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:05,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:07,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:07,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:08,965 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:10,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:10,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:10,513 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:12,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:12,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:13,966 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:15,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:15,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:16,515 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:17,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:17,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:18,993 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:19,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:04:19,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:04:19,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:04:20,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:20,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:20,563 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:04:22,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:22,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:22,517 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:24,024 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:25,741 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:04:25,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:25,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:28,519 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:28,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:28,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:29,068 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:30,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:30,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:33,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:33,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:34,096 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:34,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:04:34,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:04:34,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:04:34,521 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:35,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:35,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:38,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:38,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:39,133 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:40,523 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:40,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:40,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:42,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:42,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:44,133 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:45,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:45,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:46,525 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:47,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:47,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:49,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:49,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:04:49,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:04:49,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:04:50,430 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:04:50,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:50,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:52,527 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:52,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:52,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:54,322 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:04:55,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:55,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:57,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:57,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:04:58,529 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:04:59,350 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:00,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:00,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:04,247 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:05:04,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:04,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:04,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:04,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:05:04,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:05:04,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:05:04,531 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:06,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:06,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:09,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:09,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:09,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:10,533 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:11,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:11,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:13,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:13,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:14,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:15,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:15,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:16,535 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:17,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:17,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:19,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:05:19,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:05:19,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:05:19,622 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:05:19,623 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:20,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:20,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:21,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:05:21,555 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:05:21,556 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:05:21,556 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:05:22,537 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:05:22,537 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:22,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:22,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:25,557 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:25,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:25,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:27,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:27,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:28,539 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:30,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:30,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:31,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:32,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:32,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:34,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:05:34,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:05:34,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:05:34,541 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:35,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:35,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:36,636 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:37,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:37,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:40,543 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:41,437 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:05:41,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:41,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:42,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:44,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:44,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:46,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:46,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:46,546 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:47,617 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:49,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:49,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:49,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:05:49,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:05:49,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:05:50,215 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:05:51,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:51,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:52,548 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:05:52,654 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:53,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:53,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:55,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:55,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:57,655 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:05:58,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:58,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:05:58,550 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:00,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:00,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:02,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:02,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:02,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:04,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:06:04,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:06:04,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:06:04,552 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:04,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:04,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:07,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:07,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:08,591 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:09,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:09,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:10,557 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:12,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:12,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:13,740 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:14,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:14,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:16,559 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:18,256 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:06:18,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:18,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:18,763 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:19,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:06:19,429 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:06:19,472 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:06:20,457 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:06:21,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:21,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:22,561 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:23,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:23,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:23,799 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:26,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:26,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:28,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:28,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:28,563 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:28,827 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:31,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:31,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:33,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:33,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:33,828 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:34,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:06:34,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:06:34,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:06:34,565 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:36,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:36,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:38,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:38,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:39,561 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:40,567 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:41,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:41,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:43,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:43,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:44,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:46,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:46,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:46,569 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:48,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:48,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:49,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:06:49,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:06:49,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:06:49,652 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:06:50,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:51,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:51,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:52,571 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:53,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:53,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:55,927 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:06:56,685 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:06:56,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:56,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:58,574 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:06:59,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:06:59,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:00,969 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:01,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:01,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:02,575 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:04,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:07:04,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:07:04,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:07:04,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:04,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:06,577 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:06,583 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:06,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:06,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:08,577 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:09,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:09,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:10,578 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:11,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:11,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:11,805 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:14,579 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:14,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:14,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:16,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:16,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:17,053 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:19,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:07:19,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:07:19,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:07:19,618 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:07:19,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:19,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:20,581 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:21,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:21,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:22,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:24,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:24,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:26,583 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:26,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:26,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:28,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:29,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:29,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:31,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:31,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:32,585 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:33,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:34,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:07:34,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:07:34,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:07:35,392 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:07:35,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:35,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:37,511 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:07:37,513 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:07:37,513 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:07:37,514 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:07:37,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:07:38,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:38,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:38,514 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:38,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:40,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:40,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:43,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:43,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:44,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:44,588 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:45,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:45,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:48,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:48,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:49,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:07:49,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:07:49,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:07:49,651 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:07:49,651 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:50,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:50,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:51,591 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:53,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:53,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:54,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:07:55,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:55,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:57,593 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:07:58,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:07:58,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:00,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:00,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:00,510 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:03,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:03,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:03,595 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:08:04,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:08:04,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:08:04,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:08:05,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:05,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:05,628 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:08,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:08,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:09,597 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:08:10,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:10,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:11,573 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:13,661 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:08:13,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:13,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:15,599 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:08:16,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:16,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:16,601 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:18,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:18,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:19,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:08:19,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:08:19,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:08:20,392 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:08:21,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:21,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:21,601 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:08:22,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:23,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:23,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:26,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:26,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:27,603 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:08:27,669 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:28,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:28,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:31,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:31,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:32,699 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:33,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:33,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:33,605 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:08:34,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:08:34,430 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:08:34,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:08:35,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:35,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:37,966 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:38,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:38,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:39,606 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:08:40,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:40,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:42,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:42,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:42,966 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:44,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:44,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:47,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:47,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:47,967 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:49,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:08:49,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:08:49,431 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:08:49,649 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:08:51,328 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:08:51,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:51,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:53,650 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:54,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:54,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:55,611 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:08:56,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:56,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:58,673 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:08:59,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:59,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:08:59,613 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:01,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:01,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:03,673 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:04,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:04,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:04,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:09:04,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:09:04,431 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:09:05,614 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:06,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:06,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:09,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:09,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:09,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:11,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:11,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:11,616 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:14,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:14,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:14,733 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:16,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:16,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:17,618 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:19,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:19,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:19,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:09:19,431 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:09:19,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:09:19,575 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:09:20,576 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:21,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:21,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:23,620 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:24,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:24,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:25,789 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:26,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:26,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:29,622 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:29,968 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:09:30,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:30,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:30,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:32,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:32,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:34,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:09:34,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:09:34,431 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:09:34,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:34,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:35,623 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:36,579 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:37,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:37,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:39,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:39,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:41,625 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:41,870 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:42,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:42,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:44,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:44,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:46,892 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:47,627 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:47,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:47,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:49,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:09:49,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:09:49,431 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:09:49,630 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:09:49,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:49,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:51,905 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:52,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:52,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:53,629 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:54,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:54,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:56,906 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:09:57,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:57,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:59,631 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:09:59,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:09:59,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:01,934 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:02,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:02,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:04,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:10:04,431 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:10:04,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:10:04,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:04,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:05,633 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:07,654 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:08,496 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:10:08,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:08,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:11,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:11,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:11,635 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:12,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:13,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:13,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:16,075 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:10:16,076 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:10:16,077 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:10:16,078 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:10:16,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:16,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:16,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:10:16,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:17,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:18,078 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:18,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:18,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:19,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:10:19,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:10:19,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:10:19,573 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:10:21,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:21,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:23,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:23,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:23,639 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:24,030 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:26,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:26,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:28,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:28,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:29,071 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:29,641 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:31,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:31,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:33,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:33,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:34,072 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:34,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:10:34,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:10:34,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:10:35,643 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:36,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:36,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:38,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:38,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:39,657 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:41,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:41,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:41,644 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:43,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:43,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:45,127 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:47,125 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:10:47,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:47,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:47,646 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:49,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:10:49,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:10:49,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:10:49,654 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:10:50,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:50,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:50,656 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:52,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:52,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:53,648 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:10:55,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:55,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:56,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:10:57,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:57,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:10:59,650 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:00,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:00,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:01,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:02,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:02,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:04,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:11:04,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:11:04,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:11:05,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:05,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:05,652 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:06,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:07,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:07,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:10,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:10,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:11,654 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:12,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:12,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:12,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:15,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:15,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:17,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:17,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:17,560 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:17,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:19,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:11:19,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:11:19,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:11:20,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:20,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:20,943 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:11:22,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:22,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:22,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:23,657 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:25,700 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:11:25,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:25,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:27,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:28,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:28,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:29,659 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:30,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:30,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:32,621 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:33,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:33,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:34,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:11:34,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:11:34,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:11:35,661 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:35,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:35,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:37,645 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:38,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:38,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:40,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:40,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:41,662 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:42,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:43,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:43,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:45,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:45,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:47,707 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:48,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:48,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:49,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:11:49,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:11:49,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:11:49,664 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:50,593 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:11:50,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:50,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:52,733 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:53,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:53,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:55,666 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:11:55,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:55,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:57,734 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:11:57,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:11:57,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:00,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:00,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:01,668 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:02,776 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:04,392 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:12:04,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:04,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:04,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:12:04,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:12:04,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:12:07,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:07,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:07,670 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:07,787 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:09,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:09,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:12,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:12,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:12,812 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:13,672 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:14,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:14,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:17,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:17,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:17,840 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:19,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:12:19,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:12:19,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:12:19,624 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:12:19,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:19,674 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:19,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:22,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:22,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:22,861 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:24,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:24,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:25,676 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:27,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:27,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:27,862 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:29,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:29,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:31,677 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:32,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:32,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:32,899 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:34,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:12:34,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:12:34,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:12:34,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:34,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:37,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:37,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:37,679 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:37,928 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:39,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:39,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:42,933 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:43,282 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:12:43,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:43,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:43,681 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:46,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:46,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:47,046 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:12:47,047 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:12:47,048 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:12:47,048 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:12:47,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:12:48,048 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:48,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:48,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:48,683 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:49,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:12:49,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:12:49,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:12:49,683 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:50,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:50,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:50,510 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:12:53,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:53,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:54,001 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:12:55,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:55,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:55,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:12:58,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:58,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:12:59,002 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:00,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:00,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:01,686 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:03,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:03,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:04,014 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:04,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:13:04,432 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:13:04,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:13:05,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:05,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:07,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:07,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:07,688 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:09,051 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:10,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:10,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:12,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:12,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:13,690 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:14,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:15,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:15,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:17,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:17,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:19,072 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:19,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:13:19,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:13:19,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:13:19,575 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:13:19,693 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:20,740 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:13:20,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:20,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:21,694 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:23,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:23,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:24,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:25,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:25,695 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:25,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:27,696 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:28,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:28,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:29,696 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:29,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:30,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:30,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:32,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:32,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:34,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:13:34,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:13:34,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:13:35,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:35,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:35,582 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:35,698 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:37,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:37,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:40,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:40,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:40,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:41,700 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:42,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:42,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:45,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:45,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:45,872 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:47,702 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:47,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:47,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:49,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:13:49,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:13:49,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:13:49,583 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:13:50,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:50,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:50,879 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:52,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:52,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:53,703 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:13:55,880 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:13:55,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:55,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:59,550 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:13:59,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:59,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:13:59,705 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:00,895 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:01,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:01,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:04,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:14:04,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:14:04,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:14:04,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:04,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:05,708 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:06,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:06,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:06,628 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:09,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:09,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:11,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:11,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:11,709 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:11,948 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:14,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:14,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:16,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:16,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:16,985 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:17,712 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:19,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:14:19,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:14:19,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:14:19,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:19,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:19,555 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:14:21,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:21,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:22,027 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:23,713 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:24,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:24,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:26,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:26,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:27,028 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:29,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:29,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:29,716 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:31,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:31,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:32,061 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:34,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:14:34,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:14:34,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:14:34,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:34,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:35,718 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:37,637 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:38,027 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:14:38,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:38,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:41,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:41,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:41,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:43,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:43,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:43,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:46,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:46,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:47,721 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:48,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:48,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:48,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:49,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:14:49,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:14:49,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:14:50,866 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:14:51,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:51,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:53,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:53,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:53,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:53,723 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:14:56,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:56,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:58,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:14:58,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:58,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:14:59,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:01,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:01,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:03,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:03,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:03,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:03,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:15:03,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:15:03,279 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:15:03,279 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:15:03,726 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:15:04,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:15:04,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:15:04,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:15:04,726 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:05,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:05,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:05,727 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:08,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:08,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:08,649 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:10,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:10,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:11,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:12,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:12,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:14,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:15,356 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:15:15,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:15,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:17,730 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:18,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:18,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:19,263 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:19,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:15:19,433 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:15:19,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:15:20,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:20,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:20,309 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:15:22,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:22,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:23,733 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:24,421 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:25,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:25,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:27,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:27,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:29,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:29,735 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:30,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:30,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:32,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:32,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:34,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:15:34,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:15:34,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:15:34,584 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:35,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:35,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:35,737 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:37,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:37,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:40,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:40,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:40,502 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:41,738 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:42,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:42,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:45,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:45,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:45,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:47,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:47,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:47,740 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:49,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:15:49,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:15:49,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:15:49,628 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:15:50,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:50,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:50,629 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:53,743 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:15:53,745 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:15:53,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:53,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:56,566 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:15:56,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:56,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:58,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:58,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:15:59,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:01,578 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:01,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:01,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:03,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:03,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:04,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:16:04,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:16:04,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:16:05,746 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:06,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:06,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:06,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:09,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:09,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:11,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:11,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:11,748 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:12,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:14,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:14,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:16,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:16,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:17,659 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:17,750 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:18,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:18,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:19,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:16:19,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:16:19,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:16:19,645 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:16:21,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:21,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:22,688 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:23,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:23,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:23,752 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:26,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:26,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:27,688 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:28,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:28,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:29,754 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:31,851 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:16:31,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:31,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:32,704 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:34,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:16:34,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:16:34,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:16:35,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:35,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:35,755 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:37,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:37,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:38,587 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:39,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:39,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:41,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:41,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:41,757 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:43,758 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:44,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:44,056 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:44,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:45,759 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:46,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:46,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:48,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:48,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:49,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:49,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:16:49,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:16:49,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:16:49,760 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:50,783 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:16:50,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:50,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:51,761 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:52,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:52,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:54,785 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:55,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:55,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:57,762 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:16:57,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:57,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:59,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:16:59,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:16:59,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:01,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:01,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:03,764 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:04,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:17:04,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:17:04,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:17:05,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:05,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:05,672 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:07,766 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:08,748 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:17:08,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:08,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:10,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:10,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:11,579 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:13,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:13,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:13,768 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:15,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:15,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:17,205 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:17,276 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:17:17,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:17:17,278 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:17:17,279 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:17:17,769 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:17:18,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:18,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:19,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:17:19,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:17:19,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:17:19,769 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:20,304 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:17:20,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:20,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:22,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:22,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:22,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:25,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:25,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:25,771 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:27,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:27,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:28,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:29,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:29,772 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:29,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:32,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:32,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:33,748 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:34,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:17:34,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:17:34,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:17:34,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:34,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:35,774 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:37,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:37,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:39,574 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:39,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:39,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:41,776 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:42,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:42,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:44,655 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:45,777 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:46,469 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:17:46,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:46,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:48,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:48,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:49,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:17:49,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:17:49,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:17:49,558 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:17:50,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:51,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:51,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:51,779 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:53,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:53,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:55,347 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:17:55,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:55,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:57,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:17:58,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:17:58,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:00,871 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:00,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:00,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:01,782 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:03,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:03,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:04,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:18:04,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:18:04,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:18:05,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:05,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:06,607 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:07,785 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:08,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:08,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:10,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:10,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:11,770 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:13,786 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:13,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:13,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:15,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:15,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:17,307 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:17,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:17,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:19,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:18:19,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:18:19,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:18:19,623 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:18:19,787 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:21,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:21,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:22,624 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:24,719 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:18:24,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:24,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:25,790 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:27,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:27,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:28,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:29,791 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:29,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:29,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:32,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:32,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:33,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:34,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:18:34,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:18:34,435 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:18:34,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:34,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:35,793 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:37,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:37,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:38,644 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:39,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:39,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:41,795 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:41,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:41,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:43,867 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:44,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:44,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:45,797 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:46,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:46,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:48,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:48,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:49,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:49,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:18:49,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:18:49,435 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:18:49,622 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:18:50,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:50,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:51,799 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:53,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:53,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:54,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:18:55,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:55,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:57,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:18:58,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:18:58,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:00,488 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:01,802 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:02,397 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:19:02,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:02,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:04,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:19:04,435 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:19:04,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:19:05,628 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:05,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:05,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:07,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:07,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:09,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:09,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:09,804 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:11,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:12,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:12,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:13,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:14,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:14,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:17,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:17,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:17,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:19,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:19:19,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:19:19,435 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:19:19,807 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:19,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:19,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:20,236 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:19:22,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:22,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:22,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:23,809 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:24,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:24,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:26,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:26,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:28,105 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:29,810 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:30,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:30,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:32,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:32,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:33,566 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:34,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:34,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:34,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:19:34,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:19:34,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:19:34,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:19:34,581 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:19:34,581 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:19:34,581 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:19:34,813 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:19:35,813 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:36,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:36,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:38,582 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:39,647 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:19:39,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:39,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:41,815 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:42,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:42,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:44,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:44,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:44,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:45,817 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:47,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:47,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:49,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:49,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:19:49,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:19:49,435 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:19:49,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:49,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:50,451 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:19:51,819 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:52,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:52,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:54,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:19:54,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:54,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:56,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:56,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:57,821 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:19:59,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:19:59,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:00,110 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:02,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:02,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:03,825 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:04,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:20:04,435 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:20:04,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:20:05,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:05,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:05,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:07,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:07,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:07,828 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:09,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:09,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:10,811 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:11,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:11,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:13,830 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:13,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:13,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:16,711 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:17,744 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:20:17,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:17,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:19,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:20:19,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:20:19,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:20:19,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:20,796 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:20:20,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:20,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:21,797 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:22,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:22,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:25,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:25,835 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:26,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:27,106 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:28,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:28,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:30,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:30,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:31,837 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:32,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:32,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:32,988 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:34,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:20:34,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:20:34,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:20:35,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:35,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:35,838 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:37,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:37,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:38,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:40,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:40,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:41,840 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:42,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:42,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:44,738 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:45,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:45,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:47,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:47,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:47,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:49,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:20:49,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:20:49,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:20:50,262 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:20:50,263 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:50,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:50,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:51,844 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:52,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:52,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:55,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:20:55,791 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:20:55,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:55,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:57,846 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:20:58,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:20:58,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:00,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:00,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:01,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:03,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:03,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:03,847 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:04,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:21:04,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:21:04,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:21:05,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:05,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:06,625 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:08,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:08,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:09,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:10,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:10,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:11,902 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:13,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:13,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:15,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:15,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:15,853 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:17,504 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:18,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:18,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:19,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:21:19,435 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:21:19,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:21:19,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:20,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:20,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:20,834 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:21:22,835 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:23,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:23,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:25,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:25,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:25,857 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:28,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:28,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:28,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:29,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:30,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:30,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:33,715 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:21:33,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:33,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:33,999 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:34,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:21:34,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:21:34,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:21:35,860 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:36,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:36,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:38,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:38,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:39,850 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:41,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:41,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:41,862 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:43,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:43,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:45,513 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:46,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:46,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:47,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:48,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:48,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:49,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:21:49,435 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:21:49,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:21:50,989 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:21:50,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:51,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:51,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:51,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:21:51,462 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:21:51,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:21:51,463 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:21:51,866 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:21:51,866 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:53,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:53,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:53,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:56,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:56,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:56,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:21:57,869 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:21:58,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:21:58,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:01,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:01,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:01,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:03,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:03,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:03,870 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:04,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:22:04,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:22:04,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:22:06,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:06,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:06,657 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:08,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:08,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:09,872 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:11,667 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:22:11,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:11,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:11,979 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:13,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:14,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:14,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:16,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:16,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:17,848 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:19,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:19,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:19,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:22:19,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:22:19,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:22:19,564 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:22:19,876 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:21,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:21,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:23,566 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:24,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:24,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:25,878 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:26,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:26,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:28,947 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:29,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:29,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:29,879 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:31,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:31,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:33,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:33,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:34,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:34,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:22:34,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:22:34,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:22:35,881 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:36,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:36,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:38,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:38,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:40,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:40,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:40,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:41,883 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:43,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:43,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:45,006 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:45,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:45,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:47,885 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:49,404 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:22:49,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:49,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:49,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:22:49,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:22:49,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:22:49,628 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:22:50,629 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:51,886 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:52,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:52,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:54,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:54,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:56,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:56,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:56,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:22:57,888 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:22:59,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:22:59,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:01,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:01,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:01,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:03,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:03,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:03,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:04,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:23:04,437 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:23:04,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:23:06,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:06,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:06,618 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:07,891 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:08,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:08,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:11,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:11,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:12,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:13,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:13,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:13,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:16,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:16,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:18,075 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:18,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:18,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:19,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:23:19,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:23:19,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:23:19,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:20,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:20,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:20,680 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:23:22,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:22,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:23,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:25,897 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:26,313 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:23:26,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:26,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:28,772 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:29,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:29,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:31,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:31,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:31,900 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:34,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:34,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:34,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:34,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:23:34,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:23:34,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:23:35,902 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:36,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:36,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:39,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:39,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:39,661 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:41,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:41,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:42,904 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:44,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:44,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:45,341 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:46,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:46,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:48,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:48,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:48,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:49,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:23:49,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:23:49,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:23:49,650 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:23:50,651 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:51,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:51,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:52,907 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:23:53,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:53,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:55,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:55,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:56,052 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:23:58,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:58,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:23:58,908 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:00,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:00,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:01,944 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:03,969 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:24:04,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:04,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:04,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:24:04,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:24:04,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:24:04,910 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:06,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:06,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:07,573 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:07,645 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:24:07,646 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:24:07,647 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:24:07,649 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:24:07,911 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:24:08,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:08,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:08,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:09,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:10,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:10,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:12,650 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:14,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:14,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:14,914 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:16,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:16,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:18,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:18,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:18,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:19,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:24:19,437 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:24:19,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:24:19,602 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:24:20,915 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:21,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:21,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:23,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:23,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:23,603 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:25,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:25,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:26,917 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:28,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:28,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:29,209 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:30,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:30,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:32,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:32,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:32,919 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:34,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:34,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:24:34,436 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:24:34,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:24:34,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:34,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:36,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:36,920 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:36,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:40,047 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:40,802 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:24:40,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:40,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:42,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:42,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:42,923 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:44,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:44,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:45,940 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:47,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:47,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:48,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:49,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:24:49,437 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:24:49,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:24:49,562 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:24:49,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:49,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:51,564 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:52,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:52,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:54,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:54,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:54,927 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:57,045 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:24:57,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:57,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:58,928 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:24:59,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:24:59,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:01,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:01,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:02,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:04,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:25:04,437 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:25:04,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:25:04,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:04,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:04,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:06,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:06,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:07,659 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:08,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:08,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:10,932 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:11,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:11,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:13,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:13,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:13,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:14,933 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:17,564 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:25:17,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:17,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:18,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:19,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:25:19,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:25:19,457 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:25:19,623 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:25:20,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:20,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:20,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:22,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:22,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:23,625 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:24,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:24,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:26,937 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:27,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:27,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:28,819 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:29,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:29,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:32,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:32,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:32,940 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:34,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:34,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:34,067 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:34,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:25:34,437 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:25:34,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:25:36,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:36,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:38,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:38,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:38,941 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:39,964 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:40,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:40,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:42,943 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:43,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:43,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:45,837 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:46,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:46,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:48,944 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:49,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:49,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:49,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:25:49,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:25:49,438 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:25:49,660 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:25:51,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:51,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:51,661 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:25:54,854 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:25:54,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:54,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:54,946 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:25:56,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:56,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:25:57,634 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:00,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:00,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:00,948 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:02,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:02,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:03,420 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:04,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:26:04,438 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:26:04,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:26:05,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:05,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:06,950 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:07,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:07,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:08,604 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:10,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:10,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:10,951 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:12,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:12,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:14,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:15,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:15,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:16,953 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:17,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:17,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:19,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:19,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:26:19,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:26:19,438 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:26:19,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:19,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:20,711 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:26:22,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:22,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:22,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:24,712 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:24,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:24,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:25,731 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:26:25,733 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:26:25,733 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:26:25,734 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:26:25,956 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:26:26,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:26,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:26,957 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:27,957 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:29,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:29,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:29,735 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:32,959 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:33,231 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:26:33,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:33,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:34,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:26:34,438 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:26:34,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:26:35,597 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:36,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:36,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:38,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:38,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:38,961 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:41,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:41,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:41,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:43,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:43,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:44,962 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:45,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:45,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:47,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:48,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:48,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:48,964 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:49,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:26:49,438 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:26:49,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:26:50,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:50,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:50,835 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:26:52,837 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:53,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:53,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:54,966 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:26:55,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:55,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:57,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:26:58,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:26:58,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:00,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:00,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:00,968 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:03,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:03,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:03,718 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:04,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:27:04,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:27:04,438 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:27:04,969 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:05,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:05,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:07,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:07,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:09,639 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:10,971 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:11,346 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:27:11,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:11,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:14,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:14,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:14,640 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:16,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:16,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:16,972 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:18,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:18,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:19,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:27:19,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:27:19,438 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:27:20,987 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:27:20,987 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:21,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:21,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:22,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:23,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:23,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:25,988 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:26,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:26,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:28,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:28,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:28,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:30,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:30,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:31,198 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:32,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:33,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:33,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:34,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:27:34,438 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:27:34,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:27:35,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:35,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:36,597 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:37,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:37,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:38,979 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:39,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:39,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:42,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:42,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:42,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:44,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:44,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:44,982 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:47,714 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:48,112 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:27:48,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:48,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:48,983 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:49,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:27:49,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:27:49,439 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:27:50,556 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:27:50,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:50,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:53,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:53,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:53,557 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:27:54,984 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:27:56,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:56,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:58,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:58,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:27:58,618 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:00,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:00,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:00,986 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:02,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:02,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:03,730 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:04,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:04,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:04,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:28:04,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:28:04,439 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:28:06,987 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:07,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:07,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:09,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:09,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:09,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:12,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:12,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:12,989 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:14,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:14,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:15,046 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:16,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:17,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:17,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:19,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:19,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:19,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:28:19,439 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:28:19,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:28:19,660 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:28:20,661 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:21,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:21,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:22,992 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:25,160 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:28:25,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:25,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:25,945 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:28,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:28,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:28,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:30,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:30,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:31,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:32,995 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:33,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:33,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:34,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:28:34,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:28:34,439 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:28:35,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:35,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:36,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:38,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:38,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:38,997 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:40,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:40,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:42,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:42,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:28:42,257 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:28:42,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:28:42,258 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:28:42,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:42,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:42,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:28:43,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:44,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:45,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:45,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:47,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:47,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:47,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:49,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:49,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:49,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:28:49,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:28:49,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:28:49,629 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:28:51,000 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:52,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:52,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:52,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:54,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:54,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:56,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:56,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:57,002 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:28:58,015 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:28:59,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:28:59,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:01,003 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:02,975 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:29:03,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:03,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:03,214 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:04,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:29:04,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:29:04,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:29:05,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:05,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:07,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:07,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:07,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:09,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:09,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:09,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:11,006 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:13,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:13,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:14,887 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:15,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:15,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:17,008 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:17,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:17,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:19,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:29:19,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:29:19,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:29:19,670 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:29:20,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:20,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:20,671 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:22,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:22,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:23,010 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:25,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:25,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:26,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:27,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:27,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:29,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:29,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:31,056 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:31,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:31,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:34,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:29:34,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:29:34,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:29:34,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:34,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:36,637 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:36,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:37,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:39,015 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:40,676 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:29:40,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:40,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:41,738 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:42,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:42,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:45,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:45,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:45,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:47,636 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:47,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:47,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:49,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:29:49,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:29:49,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:29:49,617 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:29:50,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:50,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:51,019 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:52,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:53,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:53,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:29:55,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:55,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:57,021 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:29:58,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:58,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:29:59,339 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:00,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:00,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:03,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:03,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:03,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:04,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:04,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:30:04,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:30:04,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:30:05,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:05,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:07,025 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:07,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:07,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:09,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:09,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:10,060 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:12,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:12,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:13,026 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:14,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:14,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:15,904 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:17,027 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:17,658 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:30:17,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:17,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:19,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:30:19,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:30:19,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:30:20,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:20,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:21,016 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:30:21,017 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:22,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:22,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:23,029 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:25,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:25,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:26,729 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:27,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:27,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:29,031 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:29,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:29,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:32,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:33,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:33,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:34,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:30:34,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:30:34,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:30:35,032 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:35,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:35,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:37,597 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:38,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:38,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:40,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:40,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:41,034 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:42,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:42,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:43,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:44,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:45,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:45,035 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:47,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:47,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:48,559 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:49,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:30:49,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:30:49,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:30:49,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:49,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:50,858 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:30:51,037 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:51,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:51,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:53,860 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:30:55,614 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:30:55,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:55,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:57,038 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:30:58,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:58,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:30:58,963 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:00,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:00,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:01,043 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:03,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:03,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:04,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:04,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:31:04,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:31:04,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:31:05,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:05,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:07,045 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:08,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:08,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:09,830 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:10,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:10,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:11,206 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:31:11,208 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:31:11,208 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:31:11,209 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:31:12,046 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:31:12,046 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:13,047 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:13,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:13,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:15,210 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:15,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:15,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:17,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:17,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:19,048 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:19,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:31:19,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:31:19,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:31:19,605 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:31:20,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:21,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:21,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:23,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:23,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:25,050 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:25,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:25,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:26,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:27,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:27,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:29,051 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:30,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:30,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:32,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:33,592 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:31:33,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:33,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:34,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:31:34,441 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:31:34,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:31:35,053 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:36,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:36,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:37,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:38,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:38,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:40,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:40,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:41,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:42,823 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:43,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:43,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:45,056 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:45,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:45,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:47,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:47,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:48,020 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:49,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:31:49,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:31:49,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:31:50,844 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:31:50,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:50,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:52,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:53,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:53,059 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:53,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:31:55,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:55,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:57,060 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:31:57,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:57,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:31:58,933 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:01,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:01,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:03,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:03,062 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:03,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:04,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:04,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:32:04,441 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:32:04,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:32:05,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:05,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:08,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:08,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:09,063 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:09,492 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:11,928 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:32:11,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:11,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:13,065 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:13,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:14,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:14,663 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:16,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:17,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:19,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:19,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:19,066 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:19,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:32:19,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:32:19,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:32:19,658 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:32:20,659 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:21,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:21,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:24,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:24,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:25,069 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:26,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:26,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:26,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:28,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:28,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:30,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:30,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:31,071 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:32,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:34,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:34,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:34,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:32:34,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:32:34,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:32:35,072 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:36,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:36,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:37,601 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:38,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:38,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:41,074 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:41,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:41,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:43,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:43,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:43,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:46,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:46,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:47,076 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:49,025 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:49,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:32:49,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:32:49,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:32:49,659 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:32:49,853 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:32:49,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:49,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:51,077 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:53,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:53,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:54,102 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:32:55,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:55,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:57,079 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:32:57,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:32:57,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:00,021 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:00,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:00,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:02,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:02,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:03,081 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:04,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:33:04,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:33:04,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:33:04,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:04,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:05,605 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:07,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:07,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:09,083 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:09,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:09,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:11,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:12,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:12,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:14,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:14,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:15,086 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:17,202 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:17,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:17,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:19,087 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:19,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:19,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:19,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:33:19,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:33:19,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:33:19,630 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:33:21,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:21,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:22,632 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:24,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:24,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:25,089 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:27,816 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:28,278 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:33:28,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:28,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:29,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:33:29,639 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:33:29,639 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:33:29,642 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:33:30,090 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:33:30,091 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:30,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:30,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:31,090 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:33,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:33,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:33,642 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:34,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:33:34,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:33:34,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:33:35,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:35,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:37,092 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:38,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:38,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:39,635 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:40,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:40,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:43,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:43,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:43,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:45,303 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:45,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:45,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:47,095 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:47,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:47,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:49,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:33:49,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:33:49,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:33:49,616 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:33:50,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:50,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:50,617 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:52,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:52,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:53,097 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:54,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:54,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:56,208 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:33:57,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:57,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:59,099 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:33:59,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:33:59,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:02,082 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:02,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:02,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:03,100 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:04,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:34:04,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:34:04,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:34:06,106 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:34:06,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:06,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:07,669 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:08,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:08,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:09,102 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:11,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:11,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:12,960 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:13,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:13,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:15,104 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:15,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:15,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:17,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:17,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:17,962 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:19,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:34:19,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:34:19,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:34:19,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:19,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:19,596 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:34:21,106 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:22,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:22,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:23,598 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:24,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:24,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:27,108 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:27,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:27,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:28,691 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:29,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:29,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:31,109 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:32,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:32,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:34,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:34,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:34:34,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:34:34,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:34:34,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:34,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:36,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:36,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:37,111 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:38,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:38,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:39,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:42,377 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:34:42,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:42,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:43,113 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:44,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:45,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:45,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:47,114 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:47,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:47,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:49,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:34:49,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:34:49,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:34:50,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:50,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:50,793 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:34:50,793 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:52,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:52,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:53,115 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:55,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:55,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:56,333 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:34:57,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:57,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:59,117 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:34:59,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:34:59,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:02,096 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:02,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:02,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:04,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:35:04,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:35:04,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:35:04,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:04,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:05,119 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:06,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:06,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:07,662 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:09,120 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:09,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:09,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:11,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:11,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:13,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:14,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:14,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:15,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:16,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:16,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:18,857 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:19,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:35:19,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:35:19,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:35:19,579 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:35:20,235 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:35:20,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:20,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:21,124 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:23,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:23,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:24,766 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:25,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:25,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:27,127 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:27,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:27,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:29,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:30,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:30,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:32,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:32,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:33,129 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:34,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:35:34,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:35:34,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:35:35,571 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:35,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:35,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:37,130 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:37,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:37,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:40,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:40,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:40,856 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:42,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:42,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:43,132 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:45,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:45,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:45,856 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:46,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:35:46,843 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:35:46,844 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:35:46,844 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:35:47,133 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:35:47,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:47,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:48,133 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:49,134 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:49,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:35:49,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:35:49,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:35:49,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:49,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:50,222 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:35:51,223 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:53,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:53,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:53,135 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:35:55,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:55,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:56,490 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:35:58,780 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:35:58,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:58,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:35:59,138 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:01,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:01,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:02,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:03,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:03,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:04,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:36:04,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:36:04,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:36:05,140 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:06,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:06,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:07,634 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:08,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:08,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:09,141 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:11,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:11,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:13,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:13,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:13,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:15,143 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:15,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:15,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:19,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:19,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:19,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:19,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:36:19,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:36:19,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:36:19,668 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:36:21,145 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:21,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:21,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:23,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:23,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:24,670 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:26,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:26,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:27,147 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:28,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:28,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:30,012 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:30,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:31,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:33,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:33,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:33,149 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:34,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:36:34,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:36:34,444 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:36:35,648 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:36,501 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:36:36,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:36,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:37,150 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:39,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:39,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:40,994 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:41,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:41,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:43,153 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:43,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:43,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:45,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:45,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:46,802 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:47,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:47,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:49,155 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:49,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:36:49,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:36:49,444 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:36:49,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:49,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:50,790 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:36:52,596 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:52,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:52,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:54,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:54,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:55,157 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:57,596 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:36:57,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:57,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:59,158 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:36:59,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:36:59,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:01,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:01,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:03,336 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:04,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:37:04,444 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:37:04,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:37:05,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:05,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:05,160 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:07,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:07,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:08,670 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:09,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:09,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:11,161 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:12,769 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:37:12,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:12,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:14,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:15,162 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:15,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:15,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:17,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:17,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:19,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:19,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:37:19,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:37:19,445 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:37:19,598 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:37:20,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:20,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:21,164 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:22,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:22,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:25,045 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:25,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:25,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:27,166 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:27,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:27,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:30,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:30,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:30,729 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:33,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:33,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:33,167 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:34,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:37:34,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:37:34,445 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:37:35,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:35,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:36,627 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:37,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:37,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:39,169 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:40,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:40,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:41,649 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:42,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:42,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:43,171 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:45,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:45,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:47,074 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:47,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:47,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:49,173 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:49,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:37:49,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:37:49,445 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:37:49,629 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:37:50,795 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:37:50,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:50,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:52,631 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:53,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:53,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:55,174 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:37:56,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:56,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:58,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:58,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:37:58,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:37:59,176 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:01,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:01,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:03,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:03,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:03,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:04,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:38:04,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:38:04,445 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:38:04,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:38:04,647 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:38:04,647 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:38:04,647 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:38:05,178 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:38:05,178 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:06,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:06,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:07,178 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:08,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:08,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:08,648 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:11,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:11,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:11,180 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:13,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:13,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:13,726 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:16,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:16,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:17,182 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:18,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:18,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:19,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:19,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:38:19,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:38:19,445 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:38:19,669 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:38:20,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:20,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:22,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:22,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:23,185 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:25,042 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:25,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:25,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:27,186 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:29,083 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:38:29,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:29,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:30,050 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:32,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:32,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:33,188 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:34,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:34,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:34,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:38:34,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:38:34,445 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:38:35,571 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:36,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:36,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:39,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:39,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:39,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:41,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:41,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:41,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:43,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:43,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:43,191 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:45,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:45,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:46,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:47,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:47,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:49,192 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:49,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:38:49,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:38:49,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:38:49,637 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:38:50,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:50,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:51,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:52,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:52,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:54,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:54,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:55,194 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:38:57,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:57,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:57,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:38:59,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:38:59,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:01,196 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:02,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:02,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:03,362 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:04,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:39:04,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:39:04,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:39:06,018 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:39:06,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:06,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:07,198 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:08,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:08,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:09,212 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:10,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:10,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:13,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:13,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:13,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:14,792 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:15,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:15,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:17,201 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:17,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:17,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:19,201 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:19,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:39:19,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:39:19,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:39:19,687 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:39:20,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:20,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:20,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:22,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:22,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:23,203 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:25,203 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:25,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:25,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:26,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:27,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:27,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:28,204 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:30,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:30,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:31,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:32,205 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:32,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:32,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:34,206 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:34,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:39:34,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:39:34,549 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:39:34,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:34,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:36,206 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:36,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:36,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:37,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:38,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:38,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:42,208 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:42,371 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:39:42,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:42,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:42,938 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:45,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:45,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:47,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:47,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:48,028 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:48,210 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:49,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:39:49,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:39:49,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:39:49,625 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:39:50,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:50,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:52,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:52,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:53,102 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:54,212 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:39:54,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:54,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:57,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:57,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:58,103 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:39:59,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:39:59,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:00,213 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:01,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:01,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:03,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:03,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:03,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:04,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:40:04,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:40:04,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:40:05,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:40:05,349 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:40:05,350 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:40:05,350 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:40:06,215 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:40:06,215 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:06,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:06,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:07,215 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:08,351 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:08,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:08,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:11,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:11,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:13,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:13,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:13,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:14,217 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:16,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:16,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:18,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:19,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:40:19,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:40:19,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:40:19,656 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:40:20,219 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:20,861 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:40:20,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:20,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:22,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:22,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:23,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:24,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:24,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:26,221 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:26,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:26,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:28,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:30,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:30,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:32,222 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:32,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:32,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:33,580 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:34,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:40:34,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:40:34,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:40:34,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:34,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:36,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:36,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:38,224 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:38,673 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:39,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:39,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:41,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:41,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:43,760 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:44,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:44,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:44,226 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:46,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:46,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:48,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:48,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:48,837 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:49,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:40:49,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:40:49,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:40:49,617 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:40:50,227 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:50,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:50,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:52,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:52,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:53,930 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:40:56,047 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:40:56,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:56,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:56,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:40:58,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:58,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:40:58,931 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:00,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:00,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:02,231 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:04,022 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:04,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:04,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:04,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:41:04,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:41:04,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:41:06,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:06,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:08,233 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:09,111 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:09,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:09,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:11,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:11,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:13,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:13,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:14,201 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:14,235 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:16,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:16,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:18,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:18,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:19,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:19,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:41:19,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:41:19,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:41:19,617 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:41:20,236 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:20,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:20,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:23,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:23,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:24,384 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:25,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:25,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:26,238 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:28,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:28,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:29,384 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:30,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:30,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:32,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:34,277 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:41:34,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:34,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:34,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:34,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:41:34,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:41:34,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:41:37,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:37,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:38,247 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:39,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:39,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:39,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:41,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:41,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:43,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:43,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:44,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:44,644 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:46,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:46,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:48,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:48,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:49,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:41:49,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:41:49,446 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:41:50,258 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:50,890 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:41:50,891 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:51,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:51,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:53,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:53,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:56,260 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:41:56,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:56,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:56,823 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:41:58,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:41:58,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:00,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:00,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:01,917 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:02,262 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:03,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:03,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:04,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:42:04,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:42:04,447 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:42:06,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:06,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:07,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:08,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:08,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:08,263 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:11,834 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:42:11,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:11,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:12,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:14,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:14,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:16,266 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:16,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:16,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:17,506 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:19,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:42:19,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:42:19,447 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:42:19,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:19,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:20,769 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:42:21,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:21,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:22,267 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:22,770 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:24,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:24,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:26,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:26,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:28,270 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:28,675 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:28,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:29,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:32,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:32,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:33,766 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:34,272 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:34,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:34,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:34,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:42:34,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:42:34,447 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:42:36,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:36,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:36,790 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:42:36,791 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:42:36,792 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:42:36,792 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:42:37,273 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:42:38,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:38,793 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:39,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:39,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:39,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:40,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:41,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:41,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:44,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:44,275 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:44,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:44,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:46,276 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:46,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:46,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:49,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:49,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:42:49,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:42:49,447 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:42:50,009 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:42:50,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:50,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:50,277 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:50,422 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:42:53,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:53,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:54,279 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:42:54,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:42:55,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:55,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:57,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:57,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:42:59,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:00,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:00,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:02,281 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:02,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:02,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:04,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:43:04,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:43:04,447 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:43:04,639 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:05,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:05,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:07,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:07,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:08,283 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:09,714 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:10,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:11,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:13,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:13,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:14,285 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:14,802 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:15,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:15,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:17,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:17,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:19,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:19,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:19,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:43:19,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:43:19,447 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:43:19,639 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:43:20,287 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:20,640 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:21,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:21,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:23,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:23,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:25,983 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:26,289 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:27,185 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:43:27,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:27,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:30,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:30,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:31,071 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:32,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:32,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:32,291 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:34,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:43:34,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:43:34,447 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:43:35,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:35,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:36,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:37,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:37,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:38,293 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:40,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:40,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:41,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:42,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:42,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:44,295 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:44,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:44,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:46,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:47,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:47,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:49,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:43:49,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:43:49,448 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:43:49,594 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:43:49,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:49,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:50,297 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:51,595 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:51,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:51,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:54,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:54,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:56,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:43:56,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:56,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:57,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:43:59,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:43:59,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:01,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:01,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:02,301 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:02,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:04,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:44:04,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:44:04,448 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:44:05,683 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:44:05,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:05,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:07,622 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:07,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:07,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:08,303 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:10,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:10,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:12,622 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:12,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:12,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:14,305 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:14,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:14,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:17,702 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:18,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:18,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:19,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:44:19,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:44:19,448 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:44:19,596 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:44:20,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:20,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:20,307 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:22,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:22,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:23,598 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:25,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:25,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:26,309 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:28,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:28,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:28,903 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:30,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:30,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:32,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:32,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:32,311 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:33,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:34,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:44:34,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:44:34,448 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:44:35,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:35,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:37,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:37,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:38,313 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:39,074 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:39,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:39,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:42,906 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:44:42,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:42,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:44,173 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:44,315 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:45,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:45,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:47,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:47,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:49,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:49,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:44:49,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:44:49,448 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:44:49,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:49,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:50,317 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:50,502 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:44:52,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:52,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:54,504 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:44:54,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:54,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:55,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:44:55,414 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:44:55,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:44:55,415 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:44:56,319 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:44:56,319 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:56,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:56,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:57,320 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:44:59,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:44:59,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:00,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:01,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:01,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:02,321 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:04,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:04,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:04,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:45:04,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:45:04,448 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:45:05,577 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:06,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:06,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:08,323 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:08,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:08,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:11,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:11,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:11,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:14,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:14,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:14,325 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:16,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:16,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:16,624 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:19,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:45:19,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:45:19,448 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:45:19,868 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:45:19,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:19,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:20,904 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:45:21,905 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:22,327 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:23,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:23,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:25,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:25,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:27,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:27,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:27,788 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:28,329 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:29,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:29,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:32,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:32,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:32,896 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:34,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:34,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:34,331 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:34,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:45:34,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:45:34,448 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:45:37,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:37,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:37,985 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:39,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:39,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:40,333 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:42,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:42,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:42,986 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:44,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:44,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:46,335 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:46,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:46,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:48,082 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:49,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:45:49,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:45:49,449 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:45:49,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:49,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:50,639 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:45:51,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:51,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:52,337 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:53,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:53,641 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:45:53,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:57,060 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:45:57,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:57,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:45:58,339 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:45:59,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:00,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:00,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:02,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:02,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:04,341 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:04,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:04,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:46:04,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:46:04,449 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:46:05,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:05,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:07,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:07,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:09,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:09,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:09,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:10,343 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:11,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:11,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:13,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:13,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:14,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:15,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:15,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:16,346 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:17,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:17,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:19,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:46:19,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:46:19,449 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:46:19,633 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:46:19,634 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:20,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:20,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:22,348 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:22,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:22,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:24,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:24,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:25,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:27,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:27,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:28,349 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:29,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:29,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:30,728 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:33,167 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:46:33,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:33,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:34,351 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:34,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:46:34,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:46:34,449 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:46:35,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:35,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:36,663 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:38,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:38,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:40,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:40,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:40,353 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:41,926 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:42,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:42,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:45,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:45,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:46,355 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:47,023 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:47,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:47,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:49,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:46:49,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:46:49,449 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:46:50,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:50,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:50,612 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:46:52,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:52,357 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:52,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:52,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:55,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:55,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:57,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:46:57,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:57,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:58,359 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:46:59,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:46:59,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:01,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:01,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:02,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:03,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:03,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:04,361 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:04,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:47:04,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:47:04,449 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:47:07,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:07,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:07,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:10,363 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:10,639 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:47:10,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:10,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:13,578 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:13,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:13,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:15,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:15,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:16,365 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:18,675 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:18,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:18,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:19,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:47:19,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:47:19,449 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:47:20,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:20,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:20,866 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:47:22,366 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:22,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:22,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:23,764 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:26,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:26,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:27,948 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:47:27,950 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:47:27,950 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:47:27,950 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:47:28,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:28,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:28,368 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:47:28,951 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:29,368 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:30,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:30,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:30,369 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:33,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:33,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:33,952 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:34,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:47:34,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:47:34,449 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:47:35,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:35,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:36,370 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:38,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:38,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:39,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:40,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:40,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:42,372 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:43,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:43,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:45,061 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:45,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:45,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:48,374 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:49,214 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:47:49,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:49,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:49,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:47:49,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:47:49,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:47:49,654 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:47:50,655 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:51,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:51,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:54,376 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:47:54,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:54,689 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:56,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:47:56,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:47:56,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:00,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:00,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:00,378 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:01,486 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:02,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:02,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:04,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:48:04,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:48:04,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:48:05,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:05,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:06,379 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:06,598 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:07,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:07,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:10,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:10,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:11,676 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:12,381 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:12,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:12,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:14,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:14,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:16,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:16,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:16,676 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:18,383 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:19,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:48:19,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:48:19,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:48:19,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:19,667 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:48:19,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:21,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:21,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:22,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:24,385 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:24,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:24,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:27,859 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:28,410 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:48:28,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:28,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:30,386 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:31,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:31,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:32,948 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:33,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:33,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:34,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:48:34,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:48:34,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:48:35,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:35,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:36,388 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:37,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:37,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:38,042 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:40,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:40,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:42,389 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:42,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:42,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:43,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:44,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:44,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:47,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:47,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:48,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:48,391 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:49,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:48:49,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:48:49,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:48:49,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:49,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:49,632 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:48:52,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:52,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:53,338 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:54,393 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:48:54,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:54,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:57,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:57,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:58,340 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:48:59,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:48:59,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:00,394 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:01,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:01,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:03,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:04,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:04,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:49:04,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:49:04,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:49:05,743 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:49:05,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:05,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:06,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:07,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:07,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:08,397 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:08,916 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:10,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:10,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:12,398 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:12,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:12,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:14,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:15,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:15,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:17,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:17,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:18,400 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:19,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:49:19,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:49:19,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:49:19,613 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:49:20,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:20,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:20,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:22,401 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:22,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:22,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:24,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:25,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:25,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:28,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:28,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:28,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:30,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:30,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:30,778 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:32,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:32,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:34,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:34,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:34,404 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:34,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:49:34,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:49:34,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:49:36,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:37,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:37,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:39,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:39,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:40,406 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:41,978 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:42,740 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:49:42,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:42,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:45,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:45,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:46,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:49:46,133 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:49:46,133 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:49:46,135 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:49:46,408 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:49:47,136 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:47,409 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:47,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:47,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:48,409 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:49,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:49:49,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:49:49,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:49:49,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:49,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:51,013 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:49:51,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:51,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:52,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:53,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:53,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:54,410 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:49:57,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:57,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:57,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:49:59,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:49:59,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:00,412 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:02,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:02,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:02,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:04,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:04,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:04,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:50:04,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:50:04,450 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:50:06,414 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:07,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:07,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:07,670 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:09,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:09,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:10,415 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:11,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:11,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:12,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:14,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:14,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:16,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:16,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:17,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:19,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:50:19,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:50:19,451 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:50:19,981 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:50:20,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:20,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:20,855 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:50:22,856 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:22,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:23,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:25,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:25,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:28,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:28,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:28,420 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:28,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:30,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:30,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:32,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:32,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:32,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:34,094 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:34,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:50:34,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:50:34,451 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:50:35,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:35,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:36,422 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:37,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:37,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:39,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:39,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:39,705 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:42,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:42,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:42,425 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:44,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:44,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:45,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:46,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:46,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:48,426 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:49,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:50:49,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:50:49,451 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:50:49,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:49,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:49,650 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:50:51,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:51,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:51,651 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:54,428 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:50:54,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:54,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:56,873 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:50:58,290 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:50:58,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:50:58,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:00,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:00,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:00,430 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:01,953 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:02,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:02,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:04,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:51:04,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:51:04,451 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:51:05,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:05,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:06,431 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:07,031 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:07,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:07,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:09,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:09,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:12,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:12,433 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:12,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:12,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:14,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:14,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:17,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:17,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:17,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:18,435 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:19,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:51:19,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:51:19,451 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:51:19,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:19,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:20,321 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:51:22,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:22,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:22,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:24,437 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:24,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:24,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:26,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:26,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:27,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:28,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:28,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:30,438 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:31,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:31,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:32,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:34,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:51:34,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:51:34,451 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:51:35,585 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:51:35,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:35,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:36,440 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:37,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:37,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:37,738 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:40,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:40,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:42,442 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:42,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:42,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:42,841 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:45,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:45,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:47,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:47,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:47,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:48,443 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:49,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:51:49,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:51:49,451 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:51:49,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:49,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:50,800 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:51:51,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:52,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:53,801 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:51:54,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:54,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:56,446 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:51:56,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:56,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:51:59,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:00,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:00,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:02,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:02,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:02,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:04,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:04,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:52:04,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:52:04,452 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:52:05,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:05,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:07,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:07,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:08,450 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:09,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:10,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:10,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:13,975 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:52:13,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:14,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:14,451 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:14,656 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:16,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:16,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:18,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:18,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:19,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:52:19,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:52:19,452 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:52:19,637 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:52:20,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:20,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:20,455 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:20,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:23,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:23,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:24,899 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:52:24,901 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:52:24,901 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:52:24,901 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:52:25,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:25,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:52:25,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:25,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:25,902 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:26,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:28,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:28,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:30,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:30,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:30,902 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:32,458 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:33,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:33,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:34,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:52:34,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:52:34,452 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:52:35,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:35,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:36,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:38,460 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:38,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:38,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:40,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:40,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:42,021 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:42,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:42,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:44,462 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:46,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:46,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:47,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:48,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:48,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:49,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:52:49,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:52:49,452 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:52:49,625 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:52:50,464 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:51,735 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:52:51,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:51,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:52,627 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:53,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:53,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:56,466 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:52:56,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:56,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:58,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:52:58,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:52:58,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:00,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:00,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:02,467 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:03,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:03,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:03,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:04,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:53:04,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:53:04,452 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:53:05,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:05,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:07,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:07,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:08,469 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:08,516 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:09,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:09,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:11,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:11,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:13,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:14,471 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:14,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:14,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:17,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:17,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:18,608 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:19,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:53:19,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:53:19,452 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:53:20,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:20,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:20,776 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:53:22,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:22,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:22,473 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:23,777 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:24,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:24,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:26,474 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:27,633 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:53:27,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:27,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:28,787 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:30,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:30,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:32,476 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:32,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:32,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:33,896 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:34,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:53:34,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:53:34,452 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:53:34,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:34,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:36,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:36,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:38,478 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:39,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:39,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:39,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:41,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:41,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:44,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:44,479 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:44,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:44,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:46,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:46,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:49,090 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:49,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:53:49,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:53:49,453 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:53:49,628 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:53:49,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:49,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:50,481 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:51,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:51,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:53,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:53,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:54,213 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:53:55,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:55,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:56,483 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:53:59,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:59,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:53:59,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:01,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:01,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:04,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:04,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:54:04,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:54:04,453 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:54:04,486 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:04,672 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:54:04,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:04,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:06,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:06,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:09,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:09,494 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:09,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:10,488 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:11,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:11,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:14,495 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:14,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:14,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:16,490 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:16,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:16,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:18,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:18,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:19,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:54:19,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:54:19,453 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:54:19,582 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:19,630 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:54:22,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:22,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:22,491 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:24,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:24,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:24,681 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:26,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:26,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:29,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:29,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:29,493 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:29,790 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:31,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:31,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:34,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:34,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:34,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:54:34,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:54:34,453 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:54:35,495 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:35,603 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:36,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:36,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:39,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:39,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:40,992 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:41,497 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:43,246 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:54:43,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:43,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:45,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:45,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:46,087 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:47,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:47,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:47,499 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:49,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:49,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:49,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:54:49,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:54:49,453 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:54:49,655 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:54:51,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:51,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:51,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:53,501 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:54:54,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:54,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:56,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:54:56,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:56,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:57,364 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:54:57,366 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:54:57,367 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:54:57,367 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:54:57,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:54:59,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:59,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:54:59,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:01,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:01,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:01,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:04,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:55:04,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:55:04,453 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:55:04,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:04,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:05,504 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:06,648 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:06,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:06,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:08,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:08,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:11,506 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:12,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:12,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:12,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:14,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:14,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:16,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:16,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:17,507 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:17,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:19,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:55:19,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:55:19,453 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:55:19,586 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:55:19,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:19,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:19,633 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:55:22,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:22,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:22,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:23,509 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:24,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:24,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:27,511 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:27,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:27,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:27,844 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:29,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:29,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:31,512 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:31,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:31,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:33,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:33,513 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:33,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:33,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:34,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:55:34,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:55:34,453 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:55:35,514 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:35,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:35,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:37,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:37,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:38,809 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:39,515 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:39,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:39,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:42,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:42,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:44,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:44,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:44,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:45,517 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:47,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:47,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:49,279 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:49,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:55:49,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:55:49,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:55:49,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:49,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:49,688 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:55:51,519 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:52,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:52,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:54,690 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:55:56,628 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:55:56,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:56,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:57,520 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:55:58,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:55:58,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:00,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:00,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:00,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:03,522 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:03,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:03,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:04,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:56:04,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:56:04,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:56:05,590 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:06,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:06,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:08,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:08,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:09,524 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:10,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:10,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:10,645 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:12,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:12,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:14,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:15,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:15,526 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:15,751 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:17,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:17,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:19,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:19,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:19,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:56:19,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:56:19,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:56:20,512 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:56:21,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:21,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:21,513 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:23,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:23,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:23,528 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:26,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:26,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:26,947 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:27,529 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:28,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:28,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:31,948 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:31,986 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:56:32,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:32,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:33,531 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:34,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:56:34,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:56:34,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:56:35,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:35,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:37,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:37,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:37,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:39,533 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:40,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:40,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:42,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:42,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:43,160 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:44,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:44,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:45,534 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:47,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:47,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:48,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:49,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:49,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:49,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:56:49,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:56:49,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:56:49,638 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:56:51,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:51,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:51,536 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:53,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:54,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:54,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:56,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:56,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:57,538 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:56:58,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:56:59,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:56:59,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:01,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:01,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:03,540 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:03,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:04,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:57:04,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:57:04,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:57:04,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:04,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:06,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:06,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:08,642 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:10,316 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:57:10,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:10,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:11,542 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:12,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:12,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:13,643 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:14,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:57:14,812 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:57:14,812 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:57:14,812 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:57:15,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:15,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:15,543 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:57:16,543 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:17,544 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:17,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:17,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:18,814 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:19,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:57:19,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:57:19,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:57:19,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:19,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:20,573 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:57:22,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:22,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:23,546 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:23,837 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:24,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:24,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:27,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:27,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:28,940 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:29,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:29,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:29,548 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:31,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:31,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:33,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:33,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:34,044 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:34,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:57:34,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:57:34,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:57:35,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:35,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:35,550 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:38,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:38,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:39,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:40,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:40,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:41,551 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:42,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:42,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:44,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:45,840 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:57:45,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:45,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:47,553 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:48,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:48,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:49,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:49,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:57:49,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:57:49,454 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:57:50,245 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:57:50,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:50,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:53,555 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:57:53,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:53,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:54,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:55,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:55,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:57,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:58,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:57:59,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:57:59,557 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:00,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:00,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:03,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:03,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:04,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:58:04,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:58:04,455 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:58:04,673 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:05,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:05,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:05,558 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:07,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:07,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:10,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:10,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:10,643 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:11,561 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:12,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:12,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:15,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:15,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:15,643 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:17,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:17,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:17,562 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:19,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:58:19,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:58:19,455 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:58:19,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:19,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:20,617 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:58:21,618 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:23,202 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:58:23,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:23,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:23,564 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:26,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:26,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:26,867 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:28,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:28,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:29,566 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:30,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:30,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:31,981 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:33,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:33,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:34,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:58:34,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:58:34,455 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:58:35,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:35,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:35,568 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:37,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:37,635 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:38,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:40,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:40,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:41,570 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:42,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:42,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:43,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:45,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:45,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:47,572 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:47,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:47,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:48,292 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:49,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:58:49,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:58:49,455 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:58:49,645 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:58:49,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:49,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:52,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:52,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:53,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:53,573 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:58:54,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:54,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:56,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:56,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:58:58,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:58:59,575 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:00,466 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:59:00,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:00,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:03,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:03,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:03,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:04,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:59:04,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:59:04,455 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:59:05,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:05,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:05,577 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:07,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:07,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:08,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:10,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:10,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:11,579 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:12,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:12,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:13,704 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:14,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:14,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:16,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:16,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:18,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:18,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:18,799 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:19,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:59:19,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:59:19,455 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:59:19,581 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:21,087 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:59:21,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:21,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:23,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:23,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:23,915 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:25,583 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:26,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:26,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:28,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:28,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:28,915 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:31,585 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:31,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:31,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:33,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:33,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:34,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:34,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:59:34,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:59:34,455 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:59:37,459 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 21:59:37,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:37,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:37,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:39,113 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:40,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:40,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:42,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:42,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:43,588 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:44,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:44,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:44,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:47,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 21:59:47,367 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 21:59:47,368 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 21:59:47,368 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 21:59:47,589 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 21:59:47,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:48,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:48,589 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:49,370 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:49,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 21:59:49,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 21:59:49,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 21:59:49,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:50,378 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 21:59:51,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:51,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:53,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:53,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:54,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 21:59:55,591 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 21:59:56,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:56,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:58,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:58,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 21:59:59,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:01,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:01,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:01,593 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:03,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:03,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:04,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:04,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:00:04,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:00:04,455 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:00:06,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:06,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:07,595 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:08,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:08,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:09,582 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:11,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:11,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:13,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:13,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:13,597 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:14,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:16,605 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:00:16,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:16,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:19,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:00:19,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:00:19,456 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:00:19,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:19,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:19,599 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:19,671 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:00:19,672 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:21,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:21,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:24,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:24,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:24,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:25,601 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:26,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:26,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:29,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:29,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:29,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:31,602 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:31,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:31,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:34,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:00:34,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:00:34,456 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:00:34,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:34,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:35,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:36,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:36,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:37,604 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:39,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:39,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:41,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:41,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:41,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:43,606 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:44,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:44,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:46,107 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:46,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:46,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:49,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:00:49,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:00:49,456 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:00:49,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:49,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:49,608 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:50,754 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:00:51,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:51,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:51,755 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:55,006 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:00:55,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:55,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:55,610 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:00:57,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:00:57,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:57,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:59,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:00:59,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:01,611 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:02,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:02,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:02,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:04,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:01:04,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:01:04,456 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:01:04,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:04,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:07,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:07,613 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:07,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:07,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:09,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:09,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:12,423 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:12,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:12,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:13,615 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:14,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:14,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:16,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:17,496 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:18,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:19,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:01:19,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:01:19,456 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:01:19,617 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:19,665 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:01:20,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:20,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:22,666 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:23,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:23,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:25,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:25,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:25,619 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:27,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:28,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:28,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:30,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:30,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:31,621 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:33,400 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:01:33,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:33,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:33,733 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:34,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:01:34,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:01:34,456 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:01:36,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:36,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:37,624 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:38,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:38,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:38,796 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:41,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:41,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:43,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:43,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:43,626 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:43,857 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:46,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:46,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:48,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:48,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:49,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:49,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:01:49,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:01:49,456 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:01:49,628 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:51,259 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:01:51,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:51,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:53,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:53,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:53,629 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:54,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:01:55,630 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:56,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:56,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:58,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:58,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:01:59,631 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:01:59,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:01,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:01,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:01,632 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:03,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:03,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:04,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:02:04,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:02:04,456 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:02:05,072 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:05,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:02:05,145 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:02:05,146 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:02:05,146 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:02:05,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:05,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:05,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:02:06,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:07,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:08,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:08,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:10,148 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:11,636 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:12,034 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:02:12,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:12,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:14,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:14,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:15,739 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:16,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:16,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:17,638 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:19,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:19,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:19,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:02:19,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:02:19,457 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:02:20,715 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:02:21,716 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:22,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:22,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:23,639 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:24,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:24,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:26,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:27,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:27,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:29,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:29,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:29,645 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:31,923 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:32,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:32,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:34,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:34,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:34,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:02:34,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:02:34,457 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:02:36,969 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:37,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:37,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:37,647 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:39,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:39,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:42,024 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:42,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:42,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:43,650 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:44,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:44,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:47,024 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:47,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:47,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:49,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:02:49,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:02:49,457 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:02:49,652 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:49,657 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:02:51,125 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:02:51,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:51,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:52,658 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:02:53,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:53,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:55,653 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:02:55,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:55,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:58,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:58,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:02:58,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:00,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:00,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:01,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:02,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:02,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:03,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:04,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:03:04,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:03:04,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:03:04,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:04,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:07,657 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:07,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:07,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:08,220 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:09,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:09,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:11,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:11,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:13,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:13,659 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:14,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:14,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:16,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:16,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:18,326 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:19,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:03:19,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:03:19,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:03:19,660 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:03:19,662 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:19,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:19,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:21,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:21,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:23,369 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:24,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:24,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:25,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:28,370 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:28,598 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:03:28,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:28,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:30,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:30,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:31,665 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:33,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:33,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:33,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:34,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:03:34,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:03:34,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:03:35,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:35,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:37,667 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:38,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:38,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:38,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:40,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:40,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:43,516 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:43,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:43,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:43,669 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:45,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:45,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:48,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:48,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:48,562 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:49,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:03:49,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:03:49,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:03:49,661 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:03:49,670 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:50,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:50,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:53,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:53,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:53,616 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:03:55,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:55,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:55,672 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:03:58,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:58,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:03:58,617 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:00,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:00,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:01,674 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:02,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:02,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:03,660 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:04,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:04:04,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:04:04,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:04:05,854 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:04:05,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:05,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:07,677 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:08,712 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:09,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:09,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:11,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:11,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:13,678 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:13,754 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:13,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:13,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:15,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:15,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:18,793 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:18,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:18,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:19,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:04:19,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:04:19,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:04:19,662 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:04:19,680 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:20,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:21,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:23,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:23,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:23,833 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:25,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:25,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:25,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:27,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:27,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:28,833 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:30,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:30,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:31,684 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:32,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:32,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:33,874 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:34,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:04:34,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:04:34,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:04:35,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:35,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:35,981 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:04:35,982 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:04:35,982 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:04:35,983 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:04:36,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:04:36,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:37,686 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:37,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:37,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:38,984 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:40,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:40,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:43,687 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:44,377 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:04:44,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:44,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:44,970 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:46,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:46,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:49,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:49,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:49,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:04:49,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:04:49,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:04:49,664 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:04:49,689 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:50,666 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:51,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:51,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:54,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:54,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:55,691 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:04:56,052 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:04:56,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:56,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:59,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:04:59,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:01,082 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:01,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:01,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:01,692 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:04,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:05:04,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:05:04,458 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:05:04,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:04,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:06,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:06,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:06,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:07,694 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:09,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:09,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:11,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:11,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:11,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:13,696 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:14,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:14,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:16,174 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:16,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:16,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:18,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:18,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:19,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:05:19,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:05:19,459 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:05:19,698 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:20,784 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:05:21,785 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:22,178 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:05:22,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:22,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:25,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:25,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:25,700 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:27,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:27,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:27,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:30,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:30,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:31,701 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:32,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:32,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:32,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:34,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:05:34,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:05:34,459 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:05:35,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:35,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:37,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:37,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:37,307 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:37,708 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:40,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:40,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:42,308 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:42,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:42,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:43,710 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:45,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:45,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:47,333 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:47,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:47,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:49,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:05:49,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:05:49,459 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:05:49,712 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:50,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:50,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:50,886 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:05:52,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:52,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:52,887 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:05:54,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:54,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:55,714 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:05:57,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:57,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:05:58,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:00,929 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:06:00,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:00,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:01,716 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:02,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:02,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:03,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:04,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:06:04,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:06:04,459 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:06:05,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:05,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:07,718 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:07,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:07,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:08,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:10,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:10,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:12,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:12,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:13,488 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:13,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:15,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:15,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:18,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:18,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:18,488 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:19,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:06:19,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:06:19,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:06:19,721 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:20,526 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:06:21,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:21,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:23,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:23,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:23,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:25,723 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:26,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:26,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:28,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:28,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:28,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:31,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:31,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:31,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:33,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:33,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:33,591 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:34,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:06:34,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:06:34,459 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:06:35,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:35,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:37,726 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:38,289 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:06:38,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:38,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:38,633 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:41,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:41,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:43,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:43,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:43,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:43,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:45,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:45,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:48,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:48,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:48,648 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:49,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:06:49,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:06:49,459 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:06:49,730 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:50,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:50,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:50,510 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:06:53,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:53,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:54,512 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:06:55,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:55,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:55,732 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:06:58,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:58,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:06:59,715 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:00,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:00,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:01,734 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:03,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:03,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:04,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:07:04,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:07:04,459 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:07:05,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:05,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:05,596 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:06,861 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:07:06,862 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:07:06,863 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:07:06,863 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:07:07,736 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:07:07,736 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:08,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:08,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:08,736 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:10,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:10,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:10,864 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:13,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:13,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:13,738 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:15,999 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:16,756 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:07:16,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:16,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:18,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:18,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:19,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:07:19,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:07:19,459 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:07:19,672 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:07:19,739 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:21,673 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:21,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:21,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:23,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:23,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:25,741 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:26,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:26,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:27,050 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:28,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:28,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:31,743 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:31,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:31,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:32,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:33,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:33,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:34,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:07:34,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:07:34,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:07:36,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:36,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:37,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:37,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:38,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:39,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:42,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:42,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:42,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:43,747 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:44,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:44,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:47,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:47,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:47,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:49,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:49,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:49,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:07:49,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:07:49,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:07:49,673 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:07:49,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:52,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:52,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:52,675 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:55,751 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:07:55,977 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:07:56,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:56,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:58,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:07:59,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:07:59,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:01,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:01,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:01,753 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:03,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:04,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:04,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:04,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:08:04,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:08:04,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:08:06,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:06,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:07,755 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:08,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:08,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:08,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:10,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:10,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:11,756 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:13,932 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:14,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:14,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:15,758 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:16,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:16,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:17,758 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:18,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:18,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:19,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:19,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:08:19,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:08:19,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:08:19,759 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:20,542 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:08:20,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:21,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:23,761 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:24,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:24,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:25,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:26,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:26,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:29,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:29,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:29,763 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:30,817 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:31,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:31,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:33,764 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:34,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:08:34,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:08:34,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:08:34,698 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:08:34,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:34,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:35,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:36,690 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:37,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:37,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:37,766 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:39,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:39,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:41,692 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:41,767 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:42,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:42,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:44,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:44,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:47,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:47,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:47,509 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:47,769 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:49,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:08:49,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:08:49,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:08:49,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:49,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:49,677 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:08:51,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:52,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:52,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:52,938 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:54,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:54,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:57,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:57,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:57,773 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:08:57,995 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:08:59,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:08:59,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:02,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:02,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:03,775 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:03,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:04,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:09:04,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:09:04,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:09:04,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:04,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:06,671 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:09:06,673 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:09:06,673 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:09:06,674 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:09:06,776 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:09:07,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:07,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:07,777 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:08,777 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:09,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:09,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:09,675 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:12,962 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:09:12,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:13,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:13,779 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:15,515 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:15,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:15,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:17,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:17,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:19,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:09:19,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:09:19,460 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:09:19,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:20,226 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:09:20,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:20,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:21,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:22,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:22,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:23,782 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:25,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:25,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:26,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:27,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:27,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:29,784 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:30,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:30,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:32,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:32,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:32,851 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:34,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:09:34,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:09:34,461 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:09:35,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:35,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:35,785 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:37,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:37,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:38,633 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:39,786 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:40,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:40,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:42,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:42,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:44,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:45,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:45,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:45,788 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:47,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:47,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:49,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:09:49,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:09:49,461 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:09:49,691 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:09:49,692 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:50,598 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:09:50,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:50,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:51,790 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:53,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:53,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:54,930 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:09:55,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:55,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:57,793 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:09:58,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:09:58,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:00,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:00,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:00,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:01,794 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:03,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:03,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:04,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:10:04,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:10:04,461 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:10:05,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:05,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:06,591 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:07,796 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:08,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:08,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:10,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:10,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:11,891 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:12,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:12,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:13,799 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:14,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:15,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:17,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:17,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:17,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:19,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:10:19,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:10:19,461 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:10:19,686 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:10:19,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:19,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:20,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:22,687 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:22,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:22,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:23,802 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:24,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:24,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:28,145 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:10:28,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:28,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:28,380 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:29,804 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:31,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:31,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:33,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:33,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:34,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:34,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:10:34,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:10:34,461 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:10:35,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:35,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:35,806 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:37,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:37,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:39,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:39,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:40,049 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:41,808 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:42,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:42,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:44,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:44,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:45,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:47,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:47,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:47,810 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:49,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:10:49,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:10:49,461 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:10:49,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:49,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:49,684 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:10:50,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:51,811 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:52,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:52,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:54,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:54,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:56,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:10:57,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:57,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:57,813 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:10:59,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:10:59,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:01,662 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:02,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:02,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:03,815 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:04,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:11:04,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:11:04,461 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:11:05,877 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:11:05,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:05,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:06,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:08,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:08,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:10,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:10,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:10,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:12,097 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:13,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:13,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:14,818 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:15,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:15,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:17,773 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:18,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:18,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:19,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:11:19,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:11:19,461 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:11:19,685 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:11:20,820 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:20,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:21,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:23,687 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:23,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:23,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:24,207 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:11:24,209 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:11:24,209 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:11:24,210 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:11:24,821 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:11:25,821 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:25,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:25,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:26,821 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:28,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:28,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:29,124 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:30,823 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:30,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:30,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:33,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:33,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:34,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:34,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:11:34,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:11:34,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:11:35,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:35,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:38,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:38,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:39,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:40,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:40,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:43,760 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:11:43,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:43,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:44,669 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:46,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:46,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:48,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:48,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:48,829 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:49,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:11:49,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:11:49,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:11:49,687 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:11:49,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:51,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:51,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:52,830 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:11:53,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:53,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:55,000 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:11:56,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:56,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:58,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:58,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:11:58,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:00,886 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:01,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:01,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:03,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:03,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:04,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:12:04,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:12:04,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:12:04,834 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:06,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:06,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:06,680 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:08,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:08,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:10,835 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:11,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:11,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:12,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:13,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:13,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:16,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:16,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:16,837 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:18,028 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:18,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:18,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:19,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:12:19,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:12:19,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:12:20,568 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:12:20,839 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:21,609 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:12:21,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:21,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:23,569 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:24,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:24,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:26,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:26,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:26,841 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:28,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:28,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:29,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:31,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:31,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:32,843 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:33,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:33,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:34,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:34,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:12:34,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:12:34,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:12:36,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:36,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:36,844 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:38,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:38,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:40,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:40,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:41,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:42,846 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:43,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:43,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:45,777 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:46,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:46,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:48,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:48,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:48,847 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:49,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:12:49,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:12:49,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:12:49,690 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:12:50,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:50,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:51,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:52,849 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:52,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:52,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:55,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:55,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:56,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:12:58,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:12:59,239 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:12:59,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:12:59,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:02,017 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:02,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:02,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:04,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:04,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:04,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:13:04,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:13:04,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:13:04,852 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:07,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:07,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:07,599 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:09,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:09,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:10,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:12,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:12,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:12,884 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:14,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:14,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:14,855 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:16,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:16,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:18,071 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:19,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:19,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:19,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:13:19,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:13:19,462 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:13:20,857 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:21,045 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:13:21,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:21,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:23,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:23,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:24,047 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:25,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:25,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:26,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:28,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:28,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:30,029 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:30,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:30,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:30,860 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:33,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:34,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:34,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:13:34,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:13:34,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:13:35,903 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:36,861 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:37,529 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:13:37,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:37,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:40,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:40,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:41,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:42,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:42,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:42,863 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:45,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:45,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:46,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:47,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:47,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:48,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:49,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:13:49,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:13:49,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:13:49,696 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:13:50,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:50,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:51,932 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:13:51,933 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:13:51,933 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:13:51,934 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:13:52,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:52,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:52,866 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:13:52,935 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:13:53,866 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:54,866 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:55,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:55,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:57,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:57,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:13:58,868 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:13:58,888 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:00,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:00,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:02,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:02,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:04,423 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:04,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:14:04,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:14:04,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:14:04,869 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:05,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:05,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:07,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:07,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:10,044 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:10,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:10,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:10,871 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:12,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:12,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:14,873 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:15,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:15,657 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:14:15,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:15,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:18,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:18,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:19,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:14:19,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:14:19,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:14:19,695 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:14:20,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:20,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:20,696 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:20,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:23,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:23,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:25,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:25,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:25,968 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:26,876 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:28,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:28,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:30,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:30,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:31,848 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:32,878 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:33,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:33,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:34,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:14:34,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:14:34,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:14:35,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:35,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:36,880 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:37,513 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:38,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:38,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:40,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:40,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:42,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:42,882 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:43,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:43,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:45,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:45,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:48,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:48,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:48,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:48,884 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:49,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:14:49,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:14:49,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:14:49,696 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:14:50,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:50,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:52,885 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:53,386 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:14:53,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:53,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:54,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:14:56,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:56,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:58,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:58,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:14:58,887 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:14:59,974 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:01,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:01,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:03,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:03,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:04,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:15:04,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:15:04,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:15:04,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:05,597 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:06,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:06,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:08,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:08,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:08,891 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:11,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:11,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:11,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:13,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:13,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:14,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:15,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:15,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:17,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:17,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:17,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:19,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:15:19,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:15:19,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:15:20,785 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:15:20,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:20,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:20,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:22,787 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:22,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:22,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:25,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:25,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:26,897 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:27,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:27,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:28,111 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:30,982 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:15:31,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:31,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:32,898 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:33,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:33,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:33,978 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:34,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:15:34,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:15:34,463 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:15:35,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:35,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:36,900 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:38,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:38,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:39,839 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:41,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:41,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:42,902 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:43,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:43,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:45,704 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:45,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:45,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:48,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:48,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:48,904 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:49,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:15:49,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:15:49,464 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:15:50,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:50,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:51,188 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:15:51,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:52,905 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:15:53,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:53,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:55,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:55,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:56,382 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:15:58,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:58,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:15:58,907 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:00,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:00,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:02,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:03,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:03,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:04,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:16:04,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:16:04,464 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:16:04,909 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:05,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:05,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:07,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:08,089 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:16:08,090 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:16:08,091 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:16:08,093 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:16:08,770 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:16:08,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:08,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:08,910 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:16:09,911 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:10,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:11,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:11,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:13,093 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:13,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:13,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:14,913 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:16,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:16,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:18,472 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:18,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:18,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:19,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:16:19,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:16:19,464 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:16:20,550 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:16:20,915 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:21,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:21,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:23,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:23,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:23,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:26,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:26,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:26,917 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:28,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:28,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:28,605 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:30,918 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:31,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:31,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:33,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:33,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:34,296 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:34,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:16:34,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:16:34,464 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:16:36,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:36,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:36,920 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:38,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:38,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:40,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:40,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:40,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:42,922 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:42,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:42,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:45,856 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:46,229 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:16:46,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:46,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:48,924 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:48,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:49,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:49,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:16:49,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:16:49,465 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:16:49,704 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:16:51,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:51,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:51,705 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:52,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:53,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:54,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:56,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:56,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:57,576 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:16:58,927 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:16:59,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:16:59,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:01,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:01,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:03,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:03,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:03,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:04,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:17:04,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:17:04,465 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:17:04,929 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:06,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:06,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:08,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:08,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:09,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:10,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:10,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:10,931 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:13,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:13,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:14,933 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:14,962 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:15,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:15,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:18,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:18,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:19,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:17:19,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:17:19,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:17:20,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:20,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:20,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:21,336 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:17:21,337 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:23,986 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:17:24,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:24,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:24,936 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:26,811 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:26,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:26,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:28,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:28,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:30,938 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:31,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:31,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:32,490 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:33,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:33,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:34,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:17:34,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:17:34,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:17:36,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:36,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:36,940 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:38,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:38,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:38,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:41,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:41,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:42,942 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:43,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:43,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:44,224 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:46,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:46,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:46,943 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:48,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:48,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:49,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:49,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:17:49,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:17:49,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:17:50,557 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:17:51,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:51,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:52,945 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:17:53,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:53,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:55,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:17:56,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:56,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:58,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:58,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:17:58,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:00,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:01,657 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:18:01,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:01,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:04,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:18:04,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:18:04,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:18:04,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:04,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:04,949 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:06,617 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:06,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:06,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:08,950 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:09,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:09,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:11,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:11,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:12,418 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:14,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:14,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:14,952 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:16,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:16,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:18,288 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:19,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:19,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:19,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:18:19,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:18:19,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:18:19,707 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:18:20,954 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:21,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:21,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:23,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:23,996 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:18:23,997 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:18:23,997 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:18:23,998 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:18:24,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:24,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:24,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:18:24,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:25,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:26,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:26,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:28,999 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:29,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:29,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:30,957 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:31,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:31,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:33,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:33,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:34,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:34,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:18:34,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:18:34,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:18:35,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:35,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:36,958 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:39,302 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:18:39,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:39,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:39,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:42,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:42,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:42,961 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:44,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:44,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:45,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:46,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:46,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:48,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:48,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:48,963 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:49,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:18:49,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:18:49,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:18:49,709 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:18:50,710 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:51,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:51,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:52,965 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:53,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:53,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:55,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:56,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:18:56,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:58,967 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:18:59,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:18:59,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:01,825 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:02,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:02,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:02,968 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:04,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:04,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:04,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:19:04,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:19:04,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:19:07,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:07,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:07,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:08,970 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:09,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:09,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:12,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:12,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:12,674 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:14,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:14,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:14,973 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:17,691 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:19:17,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:17,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:18,114 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:18,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:19,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:19:19,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:19:19,466 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:19:19,711 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:19:20,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:20,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:22,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:22,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:23,713 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:24,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:25,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:25,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:27,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:27,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:28,984 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:30,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:30,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:30,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:32,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:32,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:34,268 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:34,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:19:34,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:19:34,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:19:35,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:35,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:36,981 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:37,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:37,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:39,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:40,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:40,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:42,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:42,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:42,983 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:45,240 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:45,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:45,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:46,985 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:47,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:47,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:49,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:19:49,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:19:49,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:19:50,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:50,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:50,835 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:19:50,837 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:52,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:52,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:52,987 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:19:55,782 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:19:55,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:55,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:56,108 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:19:58,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:58,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:19:58,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:00,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:00,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:01,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:02,991 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:03,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:03,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:04,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:20:04,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:20:04,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:20:05,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:05,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:06,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:08,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:08,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:08,993 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:10,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:10,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:12,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:12,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:13,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:13,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:15,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:15,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:17,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:18,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:18,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:18,997 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:19,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:20:19,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:20:19,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:20:19,714 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:20:20,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:20,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:22,715 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:22,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:22,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:24,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:24,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:24,999 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:27,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:27,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:28,024 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:29,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:29,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:31,001 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:33,264 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:20:33,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:33,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:33,843 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:34,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:20:34,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:20:34,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:20:36,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:36,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:37,004 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:38,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:38,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:39,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:39,710 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:20:39,711 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:20:39,712 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:20:39,713 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:20:40,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:20:40,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:40,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:41,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:42,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:43,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:43,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:44,714 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:45,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:45,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:47,009 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:47,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:47,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:49,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:20:49,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:20:49,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:20:50,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:50,646 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:20:50,647 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:50,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:52,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:52,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:53,010 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:55,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:55,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:56,507 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:20:57,013 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:20:57,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:20:57,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:00,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:00,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:02,376 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:02,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:02,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:03,014 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:04,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:21:04,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:21:04,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:21:05,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:05,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:07,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:07,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:07,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:09,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:10,688 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:21:10,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:10,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:13,017 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:13,043 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:13,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:13,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:15,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:15,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:18,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:18,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:18,885 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:19,019 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:19,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:21:19,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:21:19,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:21:19,717 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:21:20,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:20,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:23,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:23,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:24,603 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:25,021 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:25,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:25,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:27,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:27,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:30,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:30,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:30,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:31,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:32,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:32,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:34,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:21:34,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:21:34,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:21:35,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:35,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:35,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:35,611 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:37,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:37,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:39,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:39,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:41,026 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:41,081 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:41,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:42,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:44,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:44,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:46,945 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:47,028 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:48,320 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:21:48,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:48,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:49,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:21:49,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:21:49,467 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:21:50,898 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:21:51,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:51,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:52,899 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:21:53,031 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:53,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:53,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:56,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:56,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:57,032 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:21:58,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:58,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:21:58,666 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:00,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:00,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:02,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:02,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:03,035 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:04,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:04,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:22:04,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:22:04,468 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:22:05,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:05,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:07,036 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:07,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:07,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:10,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:10,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:10,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:12,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:12,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:13,038 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:15,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:15,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:16,199 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:17,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:17,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:19,040 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:19,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:22:19,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:22:19,468 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:22:20,812 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:22:20,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:20,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:21,813 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:22,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:22,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:25,042 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:26,336 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:22:26,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:26,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:26,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:29,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:29,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:31,044 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:31,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:31,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:32,018 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:34,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:22:34,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:22:34,468 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:22:34,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:34,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:35,046 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:36,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:36,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:37,595 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:39,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:39,419 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:41,048 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:41,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:41,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:42,791 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:44,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:44,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:46,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:46,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:47,049 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:47,917 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:49,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:49,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:49,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:22:49,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:22:49,468 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:22:49,722 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:22:51,051 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:51,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:51,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:53,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:22:53,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:53,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:55,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:56,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:56,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:22:56,360 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:22:56,360 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:22:56,361 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:22:57,053 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:22:57,053 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:58,053 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:22:58,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:58,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:22:59,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:00,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:00,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:03,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:04,302 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:23:04,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:04,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:04,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:04,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:23:04,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:23:04,469 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:23:07,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:07,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:09,058 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:09,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:09,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:09,906 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:11,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:11,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:13,059 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:13,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:13,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:15,326 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:16,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:16,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:18,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:18,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:19,061 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:19,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:23:19,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:23:19,468 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:23:19,723 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:23:20,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:21,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:21,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:23,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:23,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:25,063 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:26,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:26,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:26,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:28,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:28,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:29,064 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:31,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:31,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:32,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:33,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:33,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:34,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:23:34,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:23:34,469 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:23:35,067 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:36,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:36,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:37,622 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:38,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:38,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:41,068 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:42,209 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:23:42,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:42,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:42,755 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:45,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:45,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:47,071 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:47,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:47,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:47,969 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:49,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:23:49,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:23:49,468 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:23:49,724 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:23:50,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:50,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:52,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:52,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:53,073 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:53,834 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:54,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:54,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:57,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:57,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:57,074 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:23:59,641 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:23:59,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:23:59,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:02,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:02,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:03,076 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:04,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:24:04,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:24:04,469 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:24:04,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:04,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:05,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:06,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:06,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:09,078 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:09,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:09,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:11,213 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:11,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:11,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:13,079 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:14,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:14,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:16,334 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:16,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:16,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:19,082 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:19,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:24:19,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:24:19,468 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:24:19,726 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:24:19,853 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:24:19,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:19,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:21,727 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:22,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:22,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:24,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:24,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:25,084 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:26,801 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:27,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:27,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:29,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:29,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:31,086 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:32,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:32,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:32,674 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:34,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:34,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:24:34,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:24:34,469 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:24:34,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:37,089 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:37,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:37,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:38,644 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:39,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:39,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:41,090 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:42,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:42,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:44,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:44,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:44,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:47,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:47,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:47,093 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:49,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:49,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:49,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:49,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:24:49,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:24:49,470 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:24:49,727 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:24:51,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:51,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:51,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:53,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:53,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:55,168 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:24:57,097 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:24:57,226 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:24:57,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:24:57,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:00,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:00,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:00,720 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:02,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:02,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:03,098 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:04,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:25:04,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:25:04,470 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:25:04,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:04,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:06,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:07,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:07,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:07,099 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:09,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:09,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:11,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:11,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:12,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:12,379 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:25:12,380 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:25:12,381 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:25:12,381 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:25:13,101 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:25:13,102 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:14,102 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:14,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:14,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:16,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:16,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:17,382 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:19,104 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:19,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:25:19,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:25:19,470 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:25:19,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:19,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:19,729 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:25:21,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:21,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:22,730 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:24,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:24,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:25,105 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:26,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:26,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:28,053 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:29,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:29,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:31,107 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:31,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:31,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:33,591 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:34,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:25:34,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:25:34,470 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:25:34,678 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:25:34,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:34,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:35,109 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:37,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:37,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:38,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:39,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:39,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:41,111 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:42,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:42,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:44,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:44,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:44,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:47,112 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:47,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:47,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:49,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:49,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:49,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:25:49,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:25:49,470 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:25:49,690 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:49,731 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:25:52,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:52,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:53,114 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:25:54,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:54,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:55,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:25:57,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:57,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:59,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:59,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:25:59,116 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:01,078 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:01,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:01,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:03,117 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:03,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:03,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:04,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:26:04,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:26:04,470 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:26:06,632 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:06,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:06,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:08,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:08,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:09,119 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:11,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:12,064 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:26:12,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:12,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:14,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:14,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:15,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:16,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:16,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:17,828 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:19,123 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:19,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:26:19,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:26:19,470 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:26:19,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:19,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:19,732 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:26:21,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:21,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:23,734 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:24,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:24,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:25,125 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:26,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:26,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:29,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:29,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:29,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:31,127 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:31,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:31,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:34,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:34,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:26:34,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:26:34,471 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:26:34,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:34,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:36,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:36,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:37,130 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:39,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:39,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:40,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:41,132 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:41,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:41,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:44,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:44,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:45,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:46,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:46,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:47,134 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:49,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:26:49,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:26:49,471 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:26:49,593 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:26:49,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:49,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:49,735 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:26:50,736 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:52,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:52,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:53,136 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:54,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:54,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:55,948 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:26:57,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:57,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:59,138 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:26:59,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:26:59,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:01,100 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:02,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:02,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:04,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:04,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:04,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:27:04,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:27:04,471 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:27:05,141 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:06,626 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:07,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:07,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:09,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:09,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:09,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:11,878 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:11,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:11,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:13,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:13,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:15,145 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:16,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:16,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:17,609 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:18,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:18,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:19,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:27:19,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:27:19,471 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:27:19,735 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:27:21,147 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:21,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:21,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:22,737 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:23,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:23,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:25,149 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:27,445 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:27:27,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:27,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:27,767 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:29,705 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:27:29,706 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:27:29,706 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:27:29,706 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:27:30,151 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:27:30,151 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:30,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:30,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:31,151 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:32,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:32,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:33,707 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:34,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:27:34,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:27:34,471 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:27:35,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:35,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:37,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:37,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:37,153 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:38,838 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:40,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:40,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:41,156 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:42,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:42,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:44,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:45,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:45,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:47,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:47,158 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:47,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:49,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:49,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:27:49,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:27:49,471 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:27:49,737 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:27:49,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:49,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:52,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:52,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:53,160 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:54,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:54,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:54,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:27:57,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:57,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:57,162 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:27:59,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:27:59,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:00,788 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:01,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:01,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:04,165 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:04,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:28:04,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:28:04,471 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:28:05,300 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:28:05,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:05,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:06,695 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:08,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:08,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:10,167 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:10,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:10,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:12,560 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:13,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:13,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:15,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:15,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:16,170 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:18,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:18,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:18,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:19,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:28:19,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:28:19,471 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:28:19,739 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:28:20,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:20,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:20,171 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:22,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:22,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:23,740 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:24,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:24,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:26,173 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:27,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:27,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:29,310 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:29,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:29,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:32,175 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:32,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:32,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:34,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:34,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:28:34,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:28:34,472 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:28:34,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:34,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:37,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:37,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:38,178 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:39,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:39,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:39,972 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:42,179 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:43,129 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:28:43,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:43,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:45,823 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:46,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:46,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:48,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:48,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:48,181 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:49,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:28:49,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:28:49,472 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:28:50,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:50,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:51,101 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:28:51,103 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:52,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:52,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:54,184 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:28:55,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:55,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:56,885 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:28:57,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:28:57,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:00,186 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:00,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:00,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:02,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:02,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:02,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:04,188 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:04,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:29:04,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:29:04,472 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:29:05,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:05,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:07,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:07,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:07,599 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:10,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:10,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:10,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:12,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:12,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:13,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:15,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:15,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:16,192 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:17,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:17,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:18,240 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:19,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:29:19,472 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:29:19,472 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:29:19,743 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:29:20,194 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:20,496 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:29:20,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:20,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:23,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:23,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:23,744 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:25,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:25,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:26,197 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:28,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:28,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:29,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:30,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:30,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:32,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:33,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:33,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:34,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:29:34,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:29:34,473 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:29:34,622 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:35,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:35,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:37,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:37,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:38,201 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:39,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:39,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:40,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:42,203 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:42,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:42,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:44,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:44,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:45,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:46,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:29:46,414 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:29:46,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:29:46,415 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:29:47,204 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:29:47,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:47,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:48,205 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:49,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:29:49,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:29:49,473 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:29:49,744 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:29:49,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:49,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:50,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:52,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:52,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:54,206 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:29:54,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:54,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:55,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:29:58,089 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:29:58,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:29:58,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:00,208 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:00,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:00,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:01,369 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:03,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:03,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:04,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:30:04,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:30:04,473 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:30:05,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:05,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:06,210 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:06,637 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:07,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:07,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:10,211 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:10,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:10,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:12,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:12,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:12,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:15,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:15,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:16,212 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:17,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:17,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:18,046 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:19,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:30:19,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:30:19,473 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:30:20,618 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:30:20,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:20,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:22,214 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:22,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:22,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:23,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:25,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:25,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:26,215 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:27,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:27,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:28,759 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:30,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:30,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:32,217 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:32,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:32,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:34,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:34,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:30:34,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:30:34,473 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:30:36,124 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:30:36,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:36,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:38,220 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:39,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:39,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:40,374 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:41,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:41,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:43,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:43,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:44,221 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:45,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:45,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:46,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:48,222 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:48,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:48,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:49,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:30:49,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:30:49,473 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:30:50,707 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:30:50,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:50,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:51,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:53,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:53,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:54,224 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:55,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:55,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:56,762 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:30:58,225 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:30:58,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:30:58,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:00,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:00,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:02,226 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:02,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:03,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:03,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:04,227 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:04,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:31:04,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:31:04,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:31:05,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:05,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:07,644 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:08,228 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:08,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:08,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:10,228 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:10,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:10,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:12,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:13,051 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:14,181 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:31:15,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:15,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:16,230 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:17,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:17,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:18,231 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:18,941 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:19,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:31:19,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:31:19,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:31:20,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:20,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:20,605 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:31:22,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:22,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:24,233 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:24,935 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:25,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:25,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:27,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:27,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:29,950 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:30,234 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:30,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:30,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:32,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:32,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:34,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:31:34,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:31:34,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:31:35,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:35,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:35,651 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:36,236 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:37,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:37,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:40,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:40,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:40,652 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:42,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:42,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:45,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:45,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:45,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:47,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:47,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:49,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:31:49,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:31:49,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:31:49,751 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:31:50,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:50,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:50,752 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:52,240 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:53,780 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:31:53,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:53,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:55,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:31:55,302 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:31:55,302 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:31:55,303 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:31:56,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:31:56,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:56,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:31:56,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:56,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:57,242 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:31:58,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:31:58,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:01,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:01,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:01,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:03,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:03,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:04,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:32:04,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:32:04,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:32:06,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:06,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:06,612 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:08,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:32:08,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:08,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:11,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:11,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:11,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:13,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:13,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:16,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:16,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:16,690 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:18,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:18,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:18,247 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:32:19,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:32:19,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:32:19,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:32:20,477 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:32:20,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:20,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:22,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:22,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:22,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:25,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:25,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:27,696 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:27,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:27,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:28,250 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:32:31,037 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:32:31,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:31,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:32,698 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:34,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:34,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:34,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:32:34,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:32:34,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:32:36,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:36,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:37,698 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:38,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:38,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:40,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:32:40,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:41,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:43,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:43,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:43,100 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:45,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:45,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:47,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:47,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:48,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:49,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:32:49,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:32:49,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:32:49,755 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:32:50,256 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:32:50,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:50,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:52,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:52,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:53,756 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:32:55,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:55,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:57,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:57,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:32:58,757 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:00,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:00,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:02,259 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:33:02,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:02,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:03,858 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:04,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:33:04,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:33:04,474 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:33:05,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:05,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:08,712 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:33:08,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:08,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:09,621 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:11,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:11,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:12,263 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:33:13,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:13,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:15,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:16,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:16,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:18,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:18,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:19,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:33:19,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:33:19,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:33:19,756 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:33:20,775 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:21,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:21,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:22,266 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:33:23,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:23,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:25,776 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:26,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:26,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:28,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:28,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:30,268 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:33:30,864 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:30,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:30,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:32,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:33,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:34,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:33:34,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:33:34,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:33:35,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:35,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:36,603 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:37,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:37,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:40,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:40,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:42,272 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:33:42,310 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:42,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:42,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:46,245 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:33:46,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:46,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:47,311 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:49,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:49,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:49,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:33:49,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:33:49,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:33:49,758 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:33:51,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:51,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:52,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:33:52,804 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:53,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:53,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:56,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:56,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:57,805 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:33:58,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:33:58,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:00,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:00,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:02,806 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:03,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:03,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:04,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:34:04,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:34:04,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:34:04,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:34:05,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:05,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:08,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:08,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:08,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:10,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:10,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:13,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:13,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:13,801 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:14,280 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:34:15,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:15,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:18,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:18,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:18,802 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:19,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:34:19,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:34:19,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:34:19,760 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:34:20,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:20,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:23,655 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:34:23,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:23,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:24,761 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:26,284 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:34:26,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:26,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:28,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:28,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:29,936 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:31,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:31,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:33,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:33,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:34,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:34:34,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:34:34,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:34:35,762 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:36,287 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:34:36,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:36,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:38,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:38,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:40,762 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:41,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:41,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:43,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:43,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:45,763 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:46,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:46,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:48,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:48,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:49,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:34:49,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:34:49,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:34:49,761 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:34:50,291 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:34:51,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:51,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:51,763 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:53,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:53,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:56,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:56,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:56,764 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:34:58,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:34:58,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:00,295 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:35:01,472 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:35:01,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:01,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:02,562 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:04,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:04,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:04,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:35:04,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:35:04,475 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:35:06,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:06,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:07,650 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:09,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:09,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:11,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:11,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:13,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:14,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:14,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:14,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:35:16,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:16,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:18,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:18,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:19,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:19,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:35:19,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:35:19,476 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:35:19,763 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:35:21,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:21,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:22,302 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:35:23,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:23,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:23,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:25,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:25,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:28,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:28,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:28,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:30,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:30,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:33,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:33,631 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:33,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:34,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:35:34,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:35:34,476 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:35:35,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:35,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:36,306 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:35:39,124 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:35:39,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:39,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:39,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:42,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:42,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:44,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:44,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:45,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:46,309 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:35:46,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:46,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:48,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:48,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:49,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:35:49,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:35:49,476 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:35:51,314 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:35:51,317 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:51,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:52,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:54,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:54,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:56,316 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:35:56,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:56,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:58,312 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:35:58,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:35:59,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:01,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:01,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:02,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:03,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:03,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:04,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:36:04,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:36:04,476 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:36:06,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:06,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:07,292 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:08,314 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:36:08,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:08,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:11,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:11,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:12,296 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:13,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:13,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:17,022 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:36:17,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:17,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:17,298 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:19,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:36:19,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:36:19,476 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:36:19,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:19,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:20,605 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:36:21,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:21,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:22,321 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:36:22,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:24,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:24,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:26,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:27,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:27,360 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:29,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:29,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:30,324 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:36:31,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:31,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:33,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:34,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:36:34,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:36:34,476 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:36:34,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:34,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:36,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:36,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:38,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:39,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:39,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:41,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:41,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:42,327 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:36:44,472 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:44,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:44,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:46,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:46,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:49,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:49,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:49,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:36:49,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:36:49,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:36:50,905 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:36:50,906 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:51,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:51,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:52,330 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:36:54,838 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:36:54,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:54,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:55,907 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:36:57,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:57,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:59,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:36:59,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:00,907 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:02,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:02,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:04,334 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:37:04,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:37:04,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:37:04,477 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:37:04,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:04,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:06,695 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:07,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:07,419 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:09,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:09,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:11,697 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:12,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:12,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:14,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:14,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:14,336 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:37:17,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:17,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:17,338 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:19,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:19,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:19,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:37:19,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:37:19,477 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:37:20,342 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:37:21,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:21,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:22,338 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:37:22,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:23,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:23,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:26,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:26,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:27,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:28,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:28,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:32,019 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:37:32,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:32,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:32,341 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:37:32,649 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:34,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:37:34,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:37:34,477 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:37:34,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:34,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:36,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:36,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:38,343 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:37:38,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:39,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:39,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:41,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:41,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:43,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:44,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:44,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:46,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:46,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:48,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:49,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:49,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:49,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:37:49,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:37:49,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:37:49,772 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:37:51,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:51,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:53,348 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:37:54,221 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:37:54,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:54,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:56,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:56,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:59,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:59,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:37:59,950 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:01,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:01,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:01,351 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:38:04,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:04,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:04,472 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:38:04,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:38:04,521 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:38:05,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:06,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:06,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:07,353 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:38:09,436 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:38:09,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:09,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:10,725 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:12,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:12,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:14,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:14,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:15,355 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:38:15,747 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:17,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:17,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:19,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:19,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:19,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:38:19,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:38:19,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:38:19,774 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:38:20,775 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:21,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:21,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:23,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:23,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:25,776 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:26,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:26,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:28,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:28,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:29,359 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:38:31,307 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:31,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:31,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:33,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:33,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:34,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:38:34,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:38:34,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:38:36,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:36,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:36,618 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:38,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:38,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:41,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:41,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:42,345 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:43,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:43,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:43,364 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:38:46,883 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:38:46,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:46,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:47,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:49,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:38:49,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:38:49,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:38:49,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:49,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:49,775 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:38:51,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:51,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:52,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:53,370 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:38:54,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:54,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:56,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:56,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:57,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:38:59,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:38:59,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:01,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:01,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:02,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:04,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:04,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:04,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:39:04,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:39:04,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:39:06,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:06,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:07,375 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:39:08,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:09,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:09,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:11,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:11,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:13,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:14,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:14,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:16,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:16,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:17,379 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:39:18,898 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:18,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:19,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:19,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:39:19,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:39:19,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:39:19,777 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:39:21,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:21,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:24,322 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:39:24,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:24,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:24,778 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:27,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:27,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:29,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:29,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:29,780 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:31,383 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:39:32,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:32,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:34,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:34,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:34,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:39:34,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:39:34,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:39:35,621 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:36,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:36,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:38,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:38,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:40,852 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:41,387 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:39:41,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:41,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:43,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:43,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:45,853 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:46,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:46,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:48,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:48,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:49,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:39:49,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:39:49,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:39:49,778 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:39:51,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:51,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:51,780 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:53,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:53,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:55,391 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:39:56,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:56,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:57,063 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:39:58,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:39:58,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:01,560 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:40:01,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:01,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:02,064 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:04,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:04,472 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:40:04,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:40:04,478 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:40:04,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:05,394 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:40:06,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:06,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:07,626 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:09,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:09,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:11,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:11,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:12,627 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:14,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:14,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:16,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:16,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:18,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:19,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:19,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:19,397 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:40:19,472 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:40:19,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:40:19,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:40:19,780 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:40:21,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:21,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:23,782 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:24,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:24,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:26,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:26,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:28,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:28,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:29,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:29,400 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:40:31,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:31,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:33,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:33,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:34,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:34,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:40:34,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:40:34,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:40:36,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:36,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:39,632 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:40:39,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:39,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:39,712 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:42,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:42,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:43,404 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:40:44,938 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:45,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:45,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:47,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:47,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:49,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:40:49,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:40:49,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:40:49,782 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:40:50,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:50,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:50,783 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:52,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:52,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:53,407 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:40:55,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:55,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:56,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:40:57,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:57,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:59,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:40:59,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:01,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:02,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:02,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:04,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:41:04,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:41:04,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:41:04,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:04,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:06,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:06,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:06,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:07,411 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:41:09,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:09,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:11,666 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:11,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:11,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:14,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:14,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:16,894 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:17,414 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:41:18,020 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:41:18,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:18,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:19,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:41:19,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:41:19,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:41:19,784 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:41:20,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:20,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:22,785 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:22,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:22,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:25,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:25,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:27,786 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:27,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:27,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:30,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:30,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:31,418 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:41:32,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:32,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:33,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:34,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:41:34,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:41:34,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:41:35,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:35,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:37,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:37,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:38,640 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:40,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:40,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:41,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:41:42,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:42,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:43,773 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:45,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:45,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:47,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:47,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:48,774 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:49,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:41:49,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:41:49,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:41:49,785 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:41:50,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:50,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:52,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:52,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:54,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:41:55,425 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:41:55,441 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:41:55,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:55,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:58,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:58,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:41:59,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:00,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:00,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:03,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:03,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:03,427 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:42:04,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:42:04,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:42:04,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:42:04,629 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:05,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:05,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:07,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:07,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:09,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:09,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:10,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:13,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:13,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:15,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:15,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:15,438 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:17,432 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:42:17,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:17,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:19,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:42:19,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:42:19,479 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:42:19,787 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:42:19,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:19,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:20,789 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:22,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:22,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:24,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:24,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:25,435 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:42:26,769 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:27,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:27,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:29,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:29,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:31,770 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:32,848 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:42:32,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:32,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:34,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:42:34,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:42:34,480 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:42:35,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:35,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:37,495 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:37,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:37,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:39,439 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:42:40,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:40,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:42,496 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:42,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:42,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:45,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:45,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:47,441 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:42:47,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:47,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:47,769 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:49,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:42:49,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:42:49,480 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:42:49,789 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:42:50,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:50,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:52,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:52,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:52,792 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:42:55,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:55,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:57,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:57,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:42:57,792 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:00,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:00,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:01,445 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:43:02,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:02,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:03,502 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:04,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:43:04,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:43:04,480 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:43:04,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:04,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:07,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:07,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:08,795 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:09,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:43:10,295 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:43:10,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:10,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:13,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:13,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:13,796 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:15,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:15,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:18,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:18,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:18,797 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:19,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:43:19,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:43:19,480 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:43:19,791 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:43:20,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:20,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:21,451 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:43:22,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:22,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:24,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:24,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:24,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:27,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:27,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:29,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:29,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:29,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:32,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:32,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:33,454 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:43:34,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:34,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:43:34,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:43:34,480 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:43:34,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:34,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:37,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:37,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:39,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:39,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:39,703 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:42,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:42,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:44,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:44,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:45,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:45,458 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:43:47,739 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:43:47,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:47,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:49,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:43:49,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:43:49,480 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:43:50,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:50,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:51,018 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:43:51,019 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:52,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:52,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:55,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:55,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:55,460 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:43:56,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:43:57,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:43:57,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:00,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:00,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:01,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:02,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:02,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:04,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:44:04,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:44:04,480 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:44:05,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:05,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:06,655 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:07,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:07,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:09,464 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:44:10,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:10,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:11,914 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:12,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:12,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:15,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:15,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:16,915 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:17,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:17,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:19,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:44:19,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:44:19,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:44:19,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:20,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:20,489 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:44:22,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:22,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:22,612 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:23,468 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:44:25,514 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:44:25,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:25,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:27,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:28,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:28,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:30,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:30,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:31,470 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:44:33,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:33,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:33,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:34,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:44:34,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:44:34,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:44:35,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:35,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:37,472 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:44:38,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:38,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:38,776 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:40,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:40,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:43,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:43,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:43,777 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:45,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:45,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:47,475 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:44:48,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:48,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:48,812 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:49,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:44:49,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:44:49,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:44:50,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:50,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:50,488 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:44:53,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:53,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:54,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:44:55,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:55,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:58,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:58,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:44:59,478 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:44:59,563 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:00,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:00,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:03,716 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:45:03,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:03,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:04,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:45:04,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:45:04,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:45:04,706 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:06,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:06,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:08,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:08,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:09,481 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:45:09,777 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:11,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:11,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:13,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:13,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:14,778 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:16,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:16,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:18,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:18,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:19,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:45:19,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:45:19,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:45:20,880 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:45:20,880 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:21,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:21,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:23,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:23,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:23,484 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:45:25,882 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:26,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:26,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:28,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:28,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:31,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:31,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:31,487 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:45:31,608 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:33,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:33,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:34,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:45:34,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:45:34,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:45:35,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:36,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:36,672 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:38,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:38,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:41,335 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:45:41,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:41,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:41,674 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:44,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:44,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:45,490 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:45:46,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:46,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:46,717 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:49,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:49,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:49,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:45:49,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:45:49,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:45:49,799 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:45:51,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:51,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:51,801 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:53,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:53,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:55,492 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:45:55,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:55,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:56,865 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:45:58,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:45:58,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:00,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:00,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:01,866 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:03,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:03,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:04,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:46:04,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:46:04,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:46:05,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:05,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:07,099 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:08,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:08,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:09,496 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:46:10,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:10,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:12,103 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:13,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:13,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:15,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:15,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:18,085 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:18,712 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:46:18,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:18,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:19,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:46:19,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:46:19,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:46:19,499 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:46:20,917 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:46:21,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:21,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:23,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:23,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:23,918 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:26,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:26,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:28,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:28,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:28,919 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:31,503 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:46:31,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:31,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:33,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:33,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:34,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:34,475 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:46:34,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:46:34,481 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:46:36,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:36,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:38,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:38,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:39,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:41,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:41,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:43,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:43,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:43,507 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:46:44,959 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:45,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:46,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:48,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:48,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:49,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:46:49,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:46:49,482 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:46:50,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:50,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:51,028 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:46:51,029 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:52,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:52,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:55,511 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:46:56,104 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:46:56,185 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:46:56,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:56,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:58,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:46:58,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:01,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:01,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:01,105 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:03,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:03,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:04,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:47:04,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:47:04,482 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:47:05,514 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:47:05,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:05,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:06,657 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:08,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:08,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:10,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:10,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:11,658 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:13,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:13,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:15,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:15,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:16,659 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:18,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:18,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:19,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:47:19,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:47:19,482 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:47:19,519 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:47:19,804 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:47:20,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:20,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:21,805 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:23,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:23,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:25,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:25,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:27,522 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:47:27,757 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:28,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:28,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:30,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:30,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:32,757 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:34,006 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:47:34,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:34,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:34,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:47:34,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:47:34,482 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:47:36,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:37,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:38,635 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:39,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:39,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:41,529 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:47:41,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:41,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:43,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:43,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:44,497 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:46,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:46,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:48,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:48,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:49,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:47:49,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:47:49,482 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:47:49,531 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:47:50,818 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:47:50,819 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:51,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:51,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:53,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:53,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:55,820 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:47:56,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:56,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:58,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:47:58,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:00,821 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:01,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:01,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:01,536 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:48:03,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:03,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:04,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:48:04,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:48:04,482 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:48:05,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:06,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:06,710 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:08,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:08,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:11,351 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:48:11,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:11,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:11,712 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:14,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:14,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:15,541 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:48:16,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:16,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:17,624 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:18,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:18,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:19,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:48:19,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:48:19,482 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:48:19,807 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:48:21,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:21,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:22,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:23,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:23,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:25,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:25,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:27,545 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:48:27,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:27,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:28,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:30,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:30,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:32,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:32,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:33,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:34,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:48:34,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:48:34,483 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:48:35,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:35,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:37,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:37,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:38,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:39,549 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:48:40,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:40,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:42,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:42,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:43,199 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:45,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:45,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:48,199 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:48,651 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:48:48,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:48,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:49,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:48:49,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:48:49,483 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:48:49,809 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:48:51,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:51,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:53,554 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:48:53,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:53,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:53,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:48:56,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:56,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:58,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:58,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:48:58,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:01,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:01,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:03,558 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:49:03,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:03,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:04,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:04,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:49:04,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:49:04,483 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:49:06,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:06,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:08,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:08,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:09,648 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:11,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:11,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:13,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:13,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:15,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:15,562 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:49:16,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:16,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:18,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:18,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:19,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:49:19,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:49:19,483 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:49:19,810 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:49:20,811 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:20,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:20,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:22,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:22,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:25,566 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:49:26,275 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:49:26,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:26,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:26,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:29,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:29,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:31,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:31,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:31,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:34,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:34,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:34,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:49:34,483 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:49:34,483 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:49:35,569 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:49:36,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:36,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:36,819 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:38,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:38,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:40,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:40,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:41,820 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:43,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:43,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:45,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:45,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:46,820 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:48,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:48,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:49,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:49:49,483 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:49:49,483 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:49:49,574 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:49:49,811 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:49:50,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:50,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:52,813 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:53,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:53,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:55,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:55,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:57,814 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:49:58,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:58,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:49:59,577 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:50:00,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:00,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:03,084 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:03,895 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:50:03,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:03,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:04,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:50:04,483 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:50:04,483 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:50:06,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:06,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:08,694 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:08,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:08,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:09,581 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:50:11,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:11,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:13,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:13,694 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:13,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:14,912 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:50:14,915 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:50:14,915 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:50:14,916 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:50:15,582 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:50:15,583 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:50:16,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:16,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:18,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:18,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:18,917 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:19,813 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:50:21,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:21,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:23,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:50:23,163 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:50:23,164 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:50:23,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:23,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:24,309 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:25,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:50:26,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:26,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:28,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:29,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:29,310 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:31,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:31,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:33,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:33,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:34,311 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:35,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:37,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:38,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:50:38,243 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:50:39,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:39,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:50:39,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:39,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:42,424 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:50:43,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:43,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:44,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:46,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:46,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:48,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:48,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:49,594 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:50:49,818 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:50:49,821 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:51,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:51,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:53,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:50:53,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:50:53,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:50:53,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:53,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:55,596 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:50:55,702 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:50:56,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:56,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:58,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:50:58,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:00,823 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:01,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:01,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:01,598 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:03,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:03,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:05,599 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:06,089 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:06,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:06,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:07,601 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:08,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:51:08,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:51:08,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:51:08,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:08,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:11,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:11,601 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:11,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:11,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:13,602 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:13,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:13,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:16,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:16,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:16,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:17,603 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:18,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:18,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:19,816 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:51:21,983 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:51:22,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:22,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:22,375 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:23,185 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:51:23,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:51:23,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:51:23,605 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:25,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:25,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:27,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:27,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:27,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:29,607 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:30,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:30,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:32,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:32,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:32,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:35,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:35,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:35,609 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:37,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:37,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:38,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:38,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:51:38,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:51:38,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:51:40,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:40,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:41,611 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:42,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:42,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:43,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:45,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:45,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:47,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:47,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:47,613 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:48,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:49,818 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:51:50,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:50,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:52,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:52,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:53,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:51:53,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:51:53,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:51:53,616 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:51:54,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:54,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:54,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:57,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:57,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:51:59,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:51:59,617 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:00,773 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:52:00,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:00,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:02,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:02,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:04,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:05,620 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:05,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:05,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:07,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:07,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:08,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:52:08,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:52:08,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:52:09,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:10,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:10,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:11,622 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:12,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:12,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:14,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:14,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:14,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:17,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:17,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:17,624 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:19,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:19,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:19,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:19,819 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:52:22,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:22,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:23,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:52:23,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:52:23,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:52:23,626 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:24,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:24,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:25,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:27,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:27,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:29,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:29,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:29,628 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:30,491 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:32,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:32,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:34,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:34,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:35,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:35,630 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:38,144 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:52:38,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:38,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:52:38,186 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:52:38,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:38,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:52:40,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:41,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:41,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:41,632 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:43,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:43,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:45,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:46,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:46,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:47,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:48,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:48,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:49,821 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:52:50,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:51,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:51,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:53,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:52:53,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:52:53,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:53,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:52:53,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:53,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:52:56,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:56,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:56,372 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:52:58,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:58,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:52:59,639 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:00,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:01,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:01,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:03,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:03,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:03,634 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:53:03,636 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:53:03,636 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:53:03,636 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:53:03,641 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:53:04,641 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:05,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:05,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:05,641 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:06,637 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:08,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:08,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:08,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:53:08,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:53:08,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:53:10,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:10,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:11,643 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:12,587 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:13,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:13,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:16,801 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:53:16,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:16,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:17,595 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:17,646 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:18,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:18,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:19,823 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:53:21,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:21,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:22,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:23,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:53:23,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:53:23,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:53:23,648 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:23,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:23,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:26,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:26,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:27,627 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:28,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:28,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:29,650 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:31,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:31,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:32,628 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:33,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:33,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:35,652 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:36,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:36,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:37,635 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:38,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:53:38,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:53:38,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:53:38,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:38,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:41,654 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:41,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:41,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:42,670 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:43,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:43,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:46,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:46,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:47,656 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:47,683 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:48,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:48,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:49,824 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:53:51,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:51,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:52,694 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:53,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:53:53,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:53:53,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:53:53,658 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:53:55,632 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:53:55,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:55,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:57,718 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:53:58,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:58,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:53:59,661 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:00,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:00,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:02,719 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:03,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:03,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:05,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:05,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:06,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:07,731 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:08,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:54:08,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:54:08,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:54:08,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:08,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:10,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:10,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:12,665 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:12,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:12,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:12,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:15,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:15,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:17,753 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:17,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:17,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:18,667 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:19,826 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:54:19,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:19,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:22,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:22,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:23,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:23,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:54:23,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:54:23,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:54:24,669 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:24,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:24,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:27,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:27,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:28,765 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:29,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:29,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:30,671 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:33,071 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:54:33,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:33,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:33,766 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:35,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:35,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:36,673 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:37,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:37,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:38,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:54:38,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:54:38,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:54:39,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:41,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:41,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:42,675 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:43,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:43,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:44,770 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:46,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:46,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:48,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:48,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:48,677 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:49,798 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:49,828 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:54:51,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:51,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:53,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:54:53,187 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:54:53,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:54:53,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:53,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:54,679 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:54:55,321 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:54:56,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:56,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:58,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:54:58,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:00,681 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:00,813 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:01,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:01,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:03,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:03,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:05,826 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:06,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:06,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:06,683 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:08,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:55:08,188 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:55:08,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:55:08,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:08,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:11,590 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:55:11,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:11,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:11,825 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:12,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:14,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:14,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:16,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:16,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:16,850 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:18,686 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:19,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:19,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:19,829 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:55:21,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:21,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:21,855 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:23,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:55:23,188 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:55:23,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:55:24,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:24,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:24,688 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:26,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:26,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:27,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:29,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:29,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:30,690 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:31,419 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:31,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:32,850 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:33,943 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:55:33,944 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:55:33,945 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:55:33,945 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:55:34,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:34,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:34,691 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:55:36,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:36,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:36,691 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:37,946 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:38,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:55:38,188 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:55:38,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:55:39,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:39,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:41,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:41,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:42,693 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:43,876 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:44,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:44,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:46,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:46,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:48,694 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:49,831 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:55:49,833 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:49,849 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:55:49,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:49,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:52,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:52,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:53,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:55:53,188 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:55:53,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:55:54,696 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:55:54,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:54,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:55,372 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:55:57,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:57,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:59,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:55:59,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:00,702 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:00,881 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:02,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:02,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:04,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:04,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:06,703 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:06,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:06,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:06,871 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:08,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:56:08,189 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:56:08,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:56:08,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:08,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:11,886 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:11,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:11,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:12,705 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:13,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:14,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:16,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:16,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:17,881 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:18,707 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:18,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:18,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:19,832 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:56:21,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:21,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:22,905 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:23,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:56:23,189 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:56:23,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:56:23,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:23,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:24,709 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:27,268 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:56:27,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:27,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:27,909 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:30,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:30,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:30,711 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:32,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:32,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:32,909 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:35,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:35,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:36,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:37,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:37,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:38,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:38,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:56:38,189 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:56:38,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:56:40,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:40,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:42,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:42,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:42,720 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:43,934 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:44,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:44,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:46,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:46,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:48,722 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:49,834 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:56:49,835 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:49,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:49,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:51,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:52,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:53,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:56:53,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:56:53,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:56:54,724 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:56:54,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:54,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:55,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:56:57,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:57,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:59,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:56:59,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:00,726 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:00,944 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:01,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:01,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:05,401 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:57:05,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:05,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:06,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:06,944 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:08,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:57:08,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:57:08,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:08,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:08,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:57:10,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:10,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:11,962 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:12,730 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:13,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:13,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:15,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:15,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:16,974 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:18,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:18,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:18,732 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:19,835 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:57:20,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:20,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:22,656 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:23,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:57:23,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:57:23,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:57:23,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:23,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:24,734 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:25,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:25,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:26,735 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:27,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:27,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:28,085 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:28,736 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:30,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:30,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:32,738 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:33,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:33,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:33,493 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:34,739 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:35,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:35,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:37,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:37,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:38,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:57:38,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:57:38,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:57:39,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:40,741 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:40,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:40,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:44,473 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:57:44,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:44,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:44,742 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:45,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:47,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:47,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:49,370 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 22:57:49,371 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 22:57:49,372 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 22:57:49,372 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 22:57:49,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:49,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:49,743 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 22:57:49,837 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:57:50,744 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:50,838 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:52,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:52,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:53,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:57:53,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:57:53,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:57:54,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:54,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:56,330 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:57:56,746 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:57:57,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:57,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:59,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:57:59,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:01,331 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:02,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:02,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:02,748 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:04,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:04,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:06,358 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:07,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:07,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:08,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:58:08,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:58:08,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:58:08,750 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:09,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:09,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:12,331 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:12,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:12,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:14,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:14,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:14,751 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:17,375 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:17,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:17,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:19,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:19,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:19,839 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:58:20,754 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:22,840 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:22,981 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:58:23,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:23,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:23,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:58:23,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:58:23,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:58:25,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:25,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:26,755 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:27,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:27,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:28,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:30,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:30,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:32,757 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:32,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:32,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:33,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:35,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:35,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:37,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:37,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:38,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:58:38,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:58:38,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:58:38,759 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:39,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:40,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:40,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:42,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:42,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:44,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:44,761 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:45,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:45,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:47,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:47,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:49,480 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:49,840 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:58:50,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:50,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:50,763 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:52,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:52,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:53,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:58:53,190 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:58:53,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:58:55,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:58:55,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:55,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:56,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:58:57,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:58:57,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:00,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:01,209 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:59:01,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:01,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:02,767 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:04,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:04,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:05,565 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:06,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:06,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:08,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:59:08,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:59:08,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:59:08,769 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:08,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:08,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:10,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:11,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:11,318 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:13,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:14,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:14,771 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:16,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:16,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:16,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:19,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:19,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:19,841 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:59:20,773 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:21,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:21,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:21,843 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:23,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:59:23,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:59:23,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:59:24,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:24,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:26,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:26,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:26,775 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:27,641 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:28,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:28,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:30,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:31,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:32,652 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:32,776 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:34,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:34,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:36,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:36,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:37,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:38,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:59:38,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:59:38,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:59:38,779 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:39,517 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 22:59:39,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:39,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:42,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:42,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:43,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:44,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:44,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:44,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:47,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:47,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:48,683 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:49,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:49,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:49,843 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 22:59:50,784 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:52,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:52,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:53,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 22:59:53,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 22:59:53,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 22:59:54,343 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 22:59:54,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:54,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:56,786 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 22:59:57,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:57,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:59,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:59,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 22:59:59,726 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:02,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:02,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:04,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:04,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:04,755 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:04,788 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:07,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:07,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:08,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:00:08,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:00:08,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:00:09,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:09,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:10,339 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:10,790 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:11,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:11,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:14,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:14,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:15,774 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:16,792 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:18,011 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:00:18,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:18,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:19,845 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:00:19,866 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:00:19,868 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:00:19,868 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:00:19,868 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:00:20,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:20,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:20,794 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:00:20,795 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:20,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:22,795 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:22,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:22,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:23,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:00:23,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:00:23,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:00:24,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:24,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:26,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:26,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:26,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:28,796 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:29,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:29,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:31,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:31,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:31,811 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:34,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:34,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:34,798 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:36,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:36,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:36,837 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:38,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:00:38,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:00:38,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:00:39,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:39,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:40,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:41,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:41,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:42,330 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:44,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:44,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:46,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:46,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:46,803 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:47,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:48,804 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:49,846 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:00:49,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:49,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:51,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:51,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:52,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:52,830 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:00:53,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:00:53,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:00:53,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:00:54,806 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:55,513 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:00:55,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:55,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:56,806 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:00:58,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:58,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:00:58,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:00,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:00,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:02,808 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:03,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:03,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:03,931 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:05,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:05,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:06,809 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:07,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:08,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:08,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:01:08,191 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:01:08,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:01:09,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:10,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:10,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:12,812 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:12,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:12,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:14,602 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:14,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:14,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:17,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:17,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:18,814 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:19,848 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:01:19,849 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:19,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:19,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:22,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:22,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:23,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:01:23,192 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:01:23,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:01:24,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:24,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:24,815 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:25,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:27,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:27,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:28,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:29,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:29,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:31,018 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:32,846 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:01:32,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:32,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:34,818 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:35,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:35,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:36,886 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:37,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:37,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:38,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:01:38,192 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:01:38,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:01:38,819 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:40,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:40,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:42,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:42,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:42,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:44,822 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:45,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:45,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:47,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:47,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:48,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:49,849 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:01:50,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:50,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:50,823 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:52,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:52,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:53,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:01:53,193 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:01:53,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:01:53,736 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:01:54,825 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:01:55,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:55,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:57,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:57,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:01:58,738 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:00,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:00,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:00,827 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:02,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:02,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:04,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:05,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:05,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:06,829 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:07,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:07,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:08,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:02:08,193 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:02:08,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:02:10,037 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:10,680 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:02:10,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:10,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:12,831 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:13,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:13,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:15,038 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:15,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:15,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:18,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:18,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:18,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:19,850 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:02:20,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:20,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:20,852 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:22,833 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:23,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:02:23,193 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:02:23,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:02:23,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:23,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:25,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:25,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:26,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:26,509 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:02:26,510 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:02:26,511 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:02:26,511 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:02:26,835 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:02:28,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:28,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:28,835 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:30,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:30,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:31,512 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:33,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:33,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:34,837 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:35,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:35,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:37,039 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:38,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:38,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:38,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:02:38,193 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:02:38,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:02:38,838 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:40,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:40,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:42,324 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:42,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:42,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:44,840 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:44,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:44,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:47,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:48,391 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:02:48,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:48,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:48,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:49,852 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:02:51,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:51,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:52,854 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:02:53,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:02:53,193 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:02:53,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:53,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:02:53,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:54,843 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:02:56,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:56,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:58,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:58,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:02:58,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:00,845 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:00,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:00,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:02,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:02,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:04,037 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:05,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:05,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:06,847 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:07,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:07,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:08,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:03:08,194 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:03:08,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:03:09,351 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:10,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:10,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:10,848 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:12,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:12,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:14,901 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:15,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:15,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:16,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:17,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:17,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:19,854 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:03:20,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:20,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:20,596 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:22,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:22,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:22,851 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:23,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:03:23,193 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:03:23,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:03:25,657 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:03:25,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:25,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:26,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:28,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:28,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:28,853 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:30,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:30,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:32,093 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:32,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:33,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:33,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:35,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:35,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:37,886 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:38,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:38,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:38,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:03:38,193 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:03:38,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:03:38,856 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:40,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:40,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:42,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:42,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:43,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:44,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:45,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:45,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:48,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:48,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:48,578 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:48,859 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:49,855 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:03:50,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:50,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:53,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:53,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:53,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:03:53,194 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:03:53,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:03:54,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:03:54,861 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:03:55,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:55,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:58,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:03:58,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:00,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:00,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:00,314 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:00,862 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:03,611 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:04:03,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:03,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:06,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:06,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:06,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:06,863 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:08,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:04:08,194 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:04:08,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:04:08,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:08,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:10,864 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:11,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:11,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:11,819 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:13,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:13,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:16,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:16,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:16,830 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:16,866 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:18,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:18,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:19,857 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:04:21,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:21,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:22,698 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:22,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:23,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:04:23,194 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:04:23,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:04:23,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:23,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:26,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:26,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:26,869 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:28,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:28,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:28,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:31,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:31,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:32,871 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:33,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:33,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:34,102 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:36,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:36,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:38,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:04:38,194 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:04:38,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:38,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:38,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:04:38,872 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:39,329 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:41,505 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:04:41,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:41,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:41,850 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:04:41,851 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:04:41,851 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:04:41,852 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:04:41,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:04:42,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:44,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:44,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:44,853 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:46,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:46,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:48,875 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:49,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:49,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:49,858 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:04:49,859 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:51,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:51,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:53,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:04:53,194 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:04:53,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:04:54,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:54,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:54,877 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:04:55,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:04:56,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:56,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:59,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:04:59,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:00,879 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:01,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:01,338 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:01,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:04,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:04,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:06,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:06,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:06,880 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:07,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:08,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:05:08,194 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:05:08,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:05:08,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:09,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:10,881 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:11,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:11,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:13,036 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:13,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:14,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:16,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:16,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:16,884 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:18,893 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:19,337 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:05:19,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:19,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:19,860 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:05:22,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:22,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:22,886 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:23,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:05:23,194 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:05:23,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:05:24,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:24,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:24,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:26,888 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:27,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:27,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:29,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:29,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:29,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:31,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:31,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:32,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:33,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:33,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:35,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:36,891 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:36,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:36,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:38,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:05:38,195 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:05:38,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:05:39,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:39,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:41,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:41,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:41,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:42,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:43,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:43,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:46,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:46,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:46,999 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:48,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:48,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:48,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:49,861 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:05:51,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:51,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:52,863 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:53,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:05:53,195 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:05:53,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:05:53,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:53,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:54,897 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:05:56,977 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:05:57,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:57,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:58,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:05:59,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:05:59,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:00,899 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:01,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:01,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:03,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:04,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:04,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:06,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:06,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:06,901 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:08,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:06:08,195 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:06:08,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:06:09,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:09,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:09,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:10,901 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:11,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:11,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:14,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:14,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:15,110 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:16,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:16,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:16,902 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:19,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:19,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:19,863 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:06:20,345 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:21,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:21,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:22,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:23,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:06:23,195 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:06:23,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:06:24,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:24,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:25,347 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:26,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:26,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:26,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:29,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:29,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:30,956 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:31,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:31,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:32,908 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:34,553 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:06:34,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:34,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:35,957 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:37,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:37,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:38,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:06:38,195 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:06:38,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:06:38,910 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:39,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:39,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:41,345 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:42,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:42,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:44,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:44,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:44,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:46,360 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:47,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:47,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:48,913 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:49,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:49,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:49,864 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:06:51,866 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:51,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:51,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:53,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:06:53,195 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:06:53,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:06:53,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:54,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:54,914 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:06:56,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:56,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:57,367 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:06:58,011 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:06:58,013 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:06:58,013 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:06:58,014 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:06:58,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:58,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:06:58,915 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:06:58,916 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:01,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:01,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:03,015 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:03,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:03,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:04,918 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:06,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:06,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:08,187 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:08,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:07:08,195 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:07:08,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:07:08,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:08,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:10,919 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:11,990 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:07:12,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:12,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:13,601 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:15,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:15,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:16,921 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:17,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:17,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:19,358 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:19,866 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:07:19,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:19,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:20,922 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:21,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:22,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:23,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:07:23,195 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:07:23,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:07:24,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:24,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:25,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:26,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:26,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:26,924 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:29,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:29,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:30,302 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:31,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:31,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:32,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:34,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:34,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:35,788 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:36,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:36,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:38,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:07:38,196 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:07:38,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:07:38,932 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:39,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:39,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:41,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:41,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:41,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:42,933 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:44,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:44,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:46,367 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:46,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:46,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:48,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:49,638 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:07:49,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:49,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:49,867 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:07:51,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:52,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:52,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:53,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:07:53,196 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:07:53,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:07:54,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:54,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:54,937 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:57,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:07:57,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:57,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:58,938 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:07:59,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:07:59,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:02,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:02,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:02,901 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:04,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:04,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:04,939 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:07,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:07,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:08,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:08,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:08:08,196 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:08:08,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:08:09,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:09,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:10,942 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:12,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:12,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:13,620 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:14,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:14,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:16,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:16,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:16,943 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:18,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:18,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:19,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:19,869 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:08:20,945 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:21,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:21,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:23,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:08:23,196 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:08:23,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:08:23,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:23,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:25,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:26,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:27,255 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:08:27,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:27,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:30,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:30,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:30,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:32,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:32,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:32,948 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:35,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:35,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:36,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:37,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:37,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:38,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:08:38,196 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:08:38,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:08:38,950 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:40,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:40,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:42,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:42,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:42,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:44,951 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:45,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:45,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:47,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:47,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:47,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:48,952 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:49,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:49,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:49,872 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:08:51,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:51,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:52,873 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:53,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:08:53,197 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:08:53,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:08:53,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:53,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:54,954 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:08:56,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:56,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:58,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:08:58,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:08:58,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:00,956 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:01,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:01,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:04,141 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:04,958 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:05,144 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:09:05,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:05,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:08,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:08,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:08,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:09:08,196 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:09:08,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:09:09,336 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:10,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:10,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:10,959 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:13,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:13,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:15,000 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:15,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:15,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:15,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:09:15,124 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:09:15,124 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:09:15,125 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:09:15,961 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:09:16,961 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:18,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:18,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:19,873 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:09:20,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:20,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:20,866 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:22,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:22,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:22,964 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:23,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:09:23,197 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:09:23,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:09:24,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:24,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:26,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:27,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:27,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:28,966 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:29,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:29,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:32,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:32,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:32,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:32,967 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:34,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:34,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:37,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:37,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:38,078 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:38,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:09:38,197 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:09:38,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:09:38,970 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:39,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:39,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:42,946 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:09:42,971 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:42,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:43,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:43,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:45,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:45,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:47,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:47,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:48,817 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:48,973 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:49,875 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:09:50,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:50,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:52,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:52,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:53,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:09:53,197 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:09:53,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:09:54,674 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:09:54,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:09:55,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:55,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:57,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:57,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:09:58,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:00,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:00,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:00,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:02,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:02,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:04,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:05,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:05,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:06,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:07,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:07,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:08,197 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:10:08,197 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:10:08,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:10:10,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:10,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:10,981 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:11,774 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:12,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:12,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:15,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:15,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:16,982 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:17,019 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:17,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:17,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:19,876 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:10:20,915 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:10:20,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:20,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:22,883 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:22,984 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:23,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:10:23,226 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:10:23,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:10:23,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:23,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:25,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:25,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:26,985 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:28,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:28,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:28,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:30,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:30,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:32,987 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:33,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:33,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:34,333 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:35,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:35,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:38,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:10:38,226 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:10:38,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:10:38,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:38,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:39,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:39,989 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:40,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:40,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:43,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:43,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:43,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:45,021 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:45,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:45,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:48,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:48,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:49,878 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:10:49,992 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:50,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:50,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:50,879 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:53,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:53,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:53,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:10:53,226 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:10:53,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:10:55,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:55,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:55,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:10:56,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:10:58,400 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:10:58,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:58,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:10:59,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:01,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:01,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:01,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:03,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:03,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:06,000 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:06,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:06,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:06,564 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:08,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:11:08,226 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:11:08,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:08,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:11:08,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:11,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:11,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:12,355 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:13,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:13,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:16,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:16,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:17,356 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:18,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:18,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:19,880 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:11:21,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:21,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:22,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:22,882 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:23,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:23,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:23,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:11:23,226 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:11:23,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:11:26,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:26,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:28,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:28,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:28,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:28,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:31,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:31,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:33,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:33,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:33,748 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:34,009 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:36,718 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:11:36,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:36,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:38,010 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:38,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:11:38,226 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:11:38,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:11:39,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:39,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:39,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:41,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:41,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:41,794 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:11:41,795 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:11:41,796 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:11:41,796 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:11:42,013 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:11:44,013 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:44,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:44,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:44,797 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:46,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:46,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:48,014 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:49,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:49,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:49,881 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:11:49,883 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:51,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:51,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:53,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:11:53,226 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:11:53,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:11:54,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:11:54,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:54,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:55,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:11:56,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:56,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:59,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:11:59,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:00,018 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:01,102 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:01,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:01,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:04,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:04,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:06,020 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:06,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:06,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:06,866 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:08,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:12:08,227 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:12:08,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:12:08,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:08,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:10,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:10,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:12,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:12,730 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:14,220 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:12:14,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:14,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:16,023 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:16,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:17,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:18,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:19,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:19,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:19,882 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:12:21,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:21,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:22,025 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:23,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:12:23,227 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:12:23,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:12:23,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:23,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:24,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:26,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:26,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:28,026 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:28,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:28,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:29,927 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:31,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:31,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:32,028 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:33,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:33,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:34,928 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:36,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:36,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:38,030 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:38,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:12:38,227 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:12:38,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:12:38,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:38,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:40,423 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:41,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:41,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:43,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:43,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:44,032 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:45,983 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:46,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:46,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:48,033 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:48,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:48,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:49,883 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:12:51,885 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:52,058 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:12:52,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:52,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:53,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:12:53,227 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:12:53,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:12:54,034 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:12:54,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:54,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:56,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:56,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:57,862 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:12:58,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:12:58,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:00,037 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:01,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:01,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:03,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:03,721 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:03,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:04,038 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:06,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:06,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:08,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:13:08,227 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:13:08,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:13:08,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:08,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:09,372 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:10,039 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:11,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:11,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:13,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:13,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:15,202 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:16,041 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:16,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:16,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:18,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:18,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:19,885 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:13:20,886 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:21,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:21,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:22,042 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:23,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:13:23,227 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:13:23,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:13:23,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:23,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:26,044 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:26,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:26,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:26,840 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:29,910 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:13:29,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:29,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:32,046 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:32,720 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:32,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:32,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:34,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:34,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:37,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:37,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:38,048 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:38,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:38,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:13:38,227 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:13:38,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:13:39,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:39,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:42,049 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:42,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:42,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:43,356 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:44,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:44,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:47,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:47,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:48,050 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:49,218 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:49,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:49,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:49,887 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:13:52,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:52,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:53,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:13:53,227 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:13:53,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:13:54,052 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:54,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:54,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:54,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:13:56,714 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:13:56,715 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:13:56,715 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:13:56,716 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:13:57,053 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:13:57,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:57,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:58,054 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:13:59,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:59,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:13:59,717 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:02,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:02,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:04,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:04,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:04,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:05,510 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:07,563 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:14:07,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:07,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:08,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:14:08,228 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:14:08,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:14:10,057 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:10,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:10,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:11,375 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:12,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:12,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:15,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:15,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:16,059 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:17,199 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:17,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:17,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:19,888 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:14:20,060 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:20,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:20,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:22,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:22,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:22,890 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:23,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:14:23,228 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:14:23,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:14:24,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:24,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:26,063 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:26,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:27,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:28,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:29,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:29,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:31,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:31,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:32,065 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:34,584 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:34,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:34,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:36,066 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:36,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:36,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:38,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:14:38,228 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:14:38,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:14:39,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:39,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:40,369 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:41,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:41,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:42,068 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:44,977 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:14:45,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:45,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:45,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:47,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:47,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:48,070 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:49,890 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:14:49,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:49,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:50,891 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:52,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:52,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:53,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:14:53,228 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:14:53,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:14:54,073 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:14:54,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:54,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:56,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:14:57,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:57,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:59,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:14:59,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:00,074 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:01,961 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:02,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:02,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:04,076 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:04,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:04,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:06,962 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:07,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:07,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:08,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:15:08,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:15:08,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:15:09,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:09,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:10,077 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:12,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:12,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:12,357 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:14,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:14,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:16,079 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:17,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:17,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:18,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:19,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:19,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:19,891 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:15:20,081 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:22,718 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:15:22,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:22,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:23,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:15:23,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:15:23,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:15:23,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:25,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:25,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:26,082 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:27,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:27,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:28,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:30,084 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:30,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:30,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:32,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:32,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:33,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:35,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:35,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:36,086 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:37,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:37,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:38,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:15:38,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:15:38,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:15:39,347 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:40,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:40,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:42,088 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:42,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:42,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:44,574 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:44,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:45,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:47,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:47,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:48,090 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:49,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:49,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:49,893 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:15:49,895 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:51,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:51,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:53,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:15:53,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:15:53,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:15:54,091 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:54,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:54,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:55,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:15:56,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:56,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:15:58,093 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:15:59,967 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:16:00,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:00,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:01,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:02,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:02,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:04,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:04,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:04,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:07,058 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:07,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:07,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:08,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:16:08,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:16:08,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:16:09,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:09,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:10,097 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:12,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:12,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:12,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:12,736 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:16:12,738 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:16:12,738 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:16:12,738 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:16:13,098 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:16:14,099 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:14,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:14,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:17,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:17,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:17,739 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:19,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:19,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:19,895 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:16:20,101 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:22,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:22,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:22,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:23,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:16:23,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:16:23,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:16:24,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:24,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:26,102 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:27,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:27,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:28,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:29,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:29,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:32,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:32,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:32,105 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:34,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:34,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:34,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:37,382 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:16:37,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:37,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:38,106 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:38,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:16:38,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:16:38,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:16:39,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:40,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:40,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:42,108 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:42,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:42,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:44,843 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:45,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:45,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:47,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:47,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:48,110 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:49,896 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:16:49,899 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:50,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:50,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:52,111 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:16:52,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:52,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:53,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:16:53,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:16:53,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:16:55,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:55,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:55,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:16:57,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:57,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:16:58,113 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:00,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:00,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:01,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:02,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:02,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:04,115 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:05,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:05,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:07,089 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:07,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:07,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:08,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:17:08,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:17:08,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:17:09,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:09,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:10,117 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:12,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:12,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:12,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:14,118 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:15,373 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:17:15,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:15,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:17,953 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:18,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:18,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:19,897 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:17:20,120 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:20,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:20,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:23,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:23,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:23,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:23,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:17:23,229 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:17:23,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:17:25,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:25,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:26,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:27,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:27,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:28,513 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:29,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:29,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:32,123 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:32,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:32,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:34,215 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:34,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:34,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:36,125 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:37,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:37,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:38,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:17:38,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:17:38,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:17:39,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:39,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:39,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:42,126 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:42,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:42,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:44,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:44,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:45,064 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:47,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:47,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:48,129 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:49,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:49,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:49,898 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:17:50,899 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:52,800 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:17:52,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:52,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:53,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:17:53,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:17:53,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:17:54,130 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:17:55,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:55,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:56,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:17:57,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:57,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:17:58,131 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:00,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:00,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:01,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:02,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:02,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:04,134 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:05,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:05,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:06,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:07,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:07,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:08,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:18:08,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:18:08,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:18:10,135 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:10,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:10,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:12,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:12,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:12,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:14,137 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:15,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:15,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:17,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:17,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:18,099 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:19,900 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:18:20,138 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:20,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:20,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:22,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:22,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:23,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:23,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:18:23,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:18:23,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:18:25,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:25,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:26,140 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:27,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:27,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:28,502 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:29,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:18:29,444 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:18:29,445 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:18:29,445 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:18:30,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:18:30,704 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:18:30,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:30,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:32,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:33,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:33,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:34,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:35,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:35,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:38,144 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:38,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:18:38,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:18:38,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:18:38,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:38,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:40,355 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:40,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:40,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:42,146 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:43,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:43,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:45,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:45,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:45,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:48,148 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:48,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:48,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:49,902 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:18:50,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:50,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:50,903 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:53,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:53,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:53,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:18:53,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:18:53,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:18:54,150 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:18:55,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:55,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:56,420 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:18:57,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:58,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:18:58,152 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:00,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:00,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:01,502 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:02,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:03,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:04,154 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:05,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:05,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:07,344 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:08,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:19:08,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:19:08,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:19:08,353 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:19:08,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:08,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:10,156 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:11,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:11,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:12,379 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:13,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:13,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:14,157 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:16,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:16,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:18,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:18,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:18,106 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:19,903 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:19:20,160 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:20,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:20,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:23,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:23,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:23,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:23,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:19:23,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:19:23,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:19:26,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:26,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:26,162 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:28,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:28,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:28,811 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:30,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:30,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:32,163 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:32,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:33,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:34,569 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:35,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:35,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:36,164 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:37,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:37,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:38,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:19:38,230 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:19:38,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:19:40,009 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:40,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:40,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:42,166 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:42,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:42,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:45,010 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:46,264 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:19:46,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:46,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:48,169 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:49,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:49,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:49,905 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:19:50,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:51,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:51,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:52,170 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:53,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:19:53,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:19:53,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:19:53,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:54,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:56,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:56,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:56,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:19:58,171 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:19:58,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:19:58,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:00,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:00,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:01,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:03,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:03,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:04,174 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:05,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:05,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:07,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:08,175 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:08,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:20:08,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:20:08,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:20:08,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:08,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:10,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:10,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:12,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:13,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:13,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:14,178 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:15,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:15,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:17,984 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:18,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:18,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:19,906 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:20:20,179 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:20,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:20,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:23,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:23,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:20:23,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:20:23,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:20:24,060 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:20:24,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:24,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:26,181 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:27,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:27,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:28,716 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:29,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:29,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:32,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:32,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:32,183 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:34,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:34,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:34,573 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:36,184 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:36,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:36,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:38,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:20:38,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:20:38,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:20:39,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:39,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:40,349 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:41,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:41,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:42,186 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:43,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:43,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:45,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:45,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:20:45,403 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:20:45,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:20:45,404 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:20:46,187 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:20:46,188 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:46,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:46,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:48,189 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:48,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:48,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:49,908 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:20:50,909 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:51,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:51,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:52,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:53,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:20:53,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:20:53,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:20:53,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:53,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:56,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:20:56,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:56,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:58,191 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:20:58,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:20:58,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:01,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:02,137 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:21:02,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:02,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:04,193 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:05,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:05,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:06,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:07,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:07,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:08,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:21:08,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:21:08,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:21:09,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:09,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:10,195 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:11,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:11,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:12,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:14,196 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:14,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:14,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:16,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:16,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:17,730 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:19,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:19,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:19,910 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:21:20,198 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:21,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:21,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:22,912 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:23,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:21:23,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:21:23,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:21:24,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:24,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:26,200 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:26,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:26,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:28,264 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:29,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:29,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:30,201 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:31,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:31,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:34,105 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:34,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:34,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:36,202 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:36,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:36,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:38,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:21:38,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:21:38,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:21:39,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:39,582 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:21:39,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:39,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:42,204 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:42,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:42,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:44,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:44,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:45,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:47,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:47,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:48,206 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:49,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:49,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:49,911 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:21:50,912 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:52,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:52,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:53,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:21:53,232 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:21:53,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:21:54,207 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:54,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:54,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:56,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:21:57,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:57,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:58,209 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:21:59,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:21:59,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:01,502 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:02,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:02,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:04,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:04,210 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:04,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:07,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:07,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:07,360 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:08,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:22:08,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:22:08,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:22:09,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:09,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:10,212 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:11,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:11,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:12,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:13,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:13,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:14,213 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:17,182 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:22:17,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:17,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:18,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:19,912 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:22:20,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:20,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:20,214 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:22,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:22,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:23,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:23,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:22:23,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:22:23,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:22:25,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:25,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:26,217 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:27,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:27,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:28,941 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:29,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:30,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:32,218 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:32,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:32,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:34,815 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:35,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:35,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:36,220 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:37,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:37,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:38,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:22:38,232 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:22:38,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:22:40,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:40,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:40,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:42,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:42,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:42,222 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:44,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:44,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:45,513 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:46,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:46,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:48,224 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:49,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:49,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:49,914 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:22:50,915 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:51,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:51,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:52,225 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:22:53,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:22:53,232 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:22:53,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:22:55,262 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:22:55,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:55,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:56,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:22:58,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:58,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:22:58,227 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:00,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:00,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:02,062 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:02,268 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:23:02,269 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:23:02,269 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:23:02,269 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:23:03,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:23:03,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:03,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:04,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:05,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:05,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:07,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:08,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:08,231 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:23:08,232 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:23:08,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:08,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:23:10,231 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:10,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:10,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:12,363 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:13,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:13,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:14,232 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:15,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:15,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:17,738 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:18,234 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:18,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:18,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:19,921 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:23:20,235 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:20,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:21,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:22,236 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:23,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:23,340 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:23:23,340 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:23:23,341 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:23:23,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:24,236 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:24,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:28,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:28,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:28,238 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:28,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:30,238 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:30,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:30,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:32,239 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:33,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:33,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:33,936 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:36,240 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:37,042 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:23:37,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:37,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:38,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:23:38,232 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:23:38,240 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:38,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:23:39,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:40,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:40,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:42,242 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:42,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:42,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:44,242 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:44,748 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:45,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:45,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:46,243 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:47,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:47,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:49,918 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:23:49,919 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:50,244 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:50,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:50,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:52,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:52,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:52,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:53,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:23:53,233 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:23:53,260 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:23:54,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:23:55,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:23:55,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:55,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:57,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:57,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:23:58,246 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:00,248 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:00,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:00,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:00,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:02,249 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:02,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:02,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:05,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:05,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:06,250 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:06,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:07,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:07,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:08,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:24:08,232 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:24:08,251 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:08,260 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:24:10,252 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:11,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:11,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:11,741 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:13,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:13,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:14,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:15,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:24:15,170 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:24:15,170 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:24:15,171 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:24:15,254 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:24:16,254 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:16,735 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:24:16,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:16,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:17,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:18,255 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:19,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:19,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:19,919 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:24:21,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:21,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:22,257 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:22,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:23,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:24:23,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:24:23,274 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:24:24,257 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:24,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:24,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:26,258 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:26,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:26,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:27,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:29,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:29,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:30,259 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:31,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:31,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:33,260 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:33,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:34,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:34,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:35,261 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:36,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:36,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:38,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:24:38,233 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:24:38,260 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:24:38,385 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:39,262 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:39,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:39,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:41,262 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:41,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:42,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:44,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:44,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:44,180 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:45,264 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:46,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:46,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:47,265 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:48,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:48,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:49,265 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:49,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:49,921 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:24:51,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:51,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:53,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:24:53,233 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:24:53,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:24:53,266 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:55,267 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:55,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:24:55,486 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:24:55,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:55,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:57,268 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:24:58,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:24:58,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:00,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:00,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:01,112 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:01,269 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:03,270 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:03,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:03,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:05,270 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:05,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:05,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:06,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:08,233 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:25:08,511 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:25:08,512 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:25:08,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:08,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:09,272 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:10,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:10,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:11,273 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:11,929 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:13,273 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:13,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:14,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:16,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:16,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:17,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:17,335 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:19,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:19,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:19,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:19,923 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:25:21,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:21,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:21,275 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:22,739 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:22,765 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:25:22,766 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:25:22,767 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:25:22,769 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:25:23,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:25:23,233 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:25:23,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:25:23,276 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:25:24,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:24,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:24,276 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:25,276 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:26,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:26,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:27,277 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:28,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:28,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:28,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:29,277 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:30,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:30,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:33,584 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:34,305 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:25:34,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:34,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:35,279 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:37,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:37,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:38,233 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:25:38,233 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:25:38,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:25:39,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:39,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:39,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:41,280 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:42,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:42,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:44,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:44,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:45,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:45,281 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:47,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:47,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:49,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:49,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:49,924 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:25:50,925 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:51,281 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:51,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:52,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:53,233 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:25:53,233 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:25:53,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:25:54,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:54,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:56,427 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:25:56,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:56,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:57,283 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:25:58,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:25:58,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:01,283 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:01,719 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:01,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:01,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:03,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:03,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:06,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:06,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:07,284 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:07,573 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:08,233 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:26:08,233 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:26:08,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:26:08,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:08,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:11,285 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:12,162 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:26:12,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:12,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:13,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:14,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:14,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:16,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:17,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:17,286 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:19,052 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:19,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:19,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:19,926 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:26:21,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:21,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:23,233 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:26:23,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:26:23,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:26:23,287 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:23,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:24,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:24,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:26,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:26,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:28,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:28,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:29,288 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:29,687 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:31,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:31,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:33,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:33,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:35,289 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:35,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:36,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:36,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:38,233 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:26:38,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:26:38,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:26:38,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:38,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:39,290 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:41,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:41,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:41,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:43,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:43,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:45,291 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:46,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:46,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:46,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:49,927 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:26:50,010 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:26:50,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:50,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:51,292 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:52,929 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:52,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:52,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:53,233 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:26:53,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:26:53,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:26:55,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:55,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:55,293 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:26:57,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:57,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:26:58,736 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:26:59,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:00,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:01,294 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:02,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:02,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:04,593 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:04,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:04,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:07,295 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:07,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:07,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:08,233 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:27:08,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:27:08,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:27:09,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:09,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:10,024 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:11,296 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:12,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:12,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:14,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:14,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:15,025 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:17,297 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:17,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:17,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:19,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:19,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:19,929 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:27:20,650 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:22,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:22,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:23,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:27:23,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:27:23,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:27:23,298 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:24,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:24,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:26,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:27,843 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:27:27,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:27,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:29,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:30,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:30,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:31,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:32,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:27:32,431 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:27:32,431 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:27:32,433 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:27:32,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:32,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:33,300 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:27:34,300 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:35,301 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:35,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:35,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:37,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:37,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:37,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:38,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:27:38,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:27:38,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:27:40,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:40,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:41,302 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:42,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:42,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:43,377 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:45,302 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:45,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:45,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:47,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:47,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:48,902 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:49,931 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:27:50,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:50,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:51,303 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:52,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:52,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:53,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:27:53,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:27:53,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:27:54,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:27:55,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:55,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:57,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:57,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:27:57,304 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:27:59,691 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:00,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:00,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:01,305 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:02,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:02,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:04,693 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:05,491 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:28:05,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:05,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:07,306 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:08,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:08,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:28:08,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:28:08,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:08,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:28:10,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:10,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:10,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:13,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:13,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:13,307 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:15,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:15,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:15,934 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:17,308 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:18,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:18,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:19,933 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:28:20,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:20,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:20,935 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:23,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:23,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:23,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:28:23,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:28:23,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:28:23,309 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:25,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:25,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:26,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:27,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:27,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:29,310 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:29,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:29,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:32,078 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:32,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:32,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:34,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:34,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:35,311 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:37,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:37,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:37,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:38,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:28:38,234 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:28:38,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:28:39,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:39,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:41,312 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:43,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:43,239 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:28:43,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:43,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:45,313 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:46,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:46,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:48,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:48,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:48,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:49,935 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:28:50,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:50,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:51,314 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:52,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:52,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:53,201 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:28:53,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:28:53,235 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:28:53,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:28:55,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:55,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:57,315 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:28:57,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:57,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:28:58,484 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:00,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:00,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:01,316 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:02,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:02,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:04,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:05,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:05,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:07,317 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:07,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:07,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:08,234 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:29:08,235 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:29:08,263 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:29:10,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:10,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:10,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:12,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:12,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:15,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:15,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:15,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:17,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:17,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:19,936 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:29:20,744 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:21,132 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:29:21,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:21,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:23,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:29:23,235 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:29:23,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:29:23,320 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:23,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:23,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:25,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:25,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:26,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:28,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:28,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:29,321 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:30,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:30,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:31,559 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:33,322 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:33,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:33,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:35,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:35,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:37,092 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:38,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:29:38,235 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:29:38,263 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:29:38,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:38,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:39,323 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:40,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:40,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:42,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:43,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:43,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:45,324 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:45,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:45,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:47,969 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:48,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:48,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:49,325 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:49,937 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:29:50,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:50,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:53,202 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:53,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:29:53,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:29:53,263 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:29:53,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:53,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:55,326 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:29:55,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:55,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:58,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:29:58,611 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:29:58,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:58,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:29:59,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:29:59,483 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:29:59,483 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:29:59,484 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:30:00,327 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:30:01,327 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:01,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:01,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:03,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:03,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:04,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:05,328 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:06,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:06,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:08,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:30:08,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:30:08,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:30:08,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:08,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:09,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:11,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:11,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:11,329 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:13,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:13,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:15,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:16,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:16,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:17,330 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:18,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:18,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:19,939 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:30:20,767 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:21,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:21,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:23,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:23,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:23,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:30:23,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:30:23,263 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:30:23,331 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:25,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:25,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:26,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:27,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:27,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:29,334 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:30,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:30,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:31,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:32,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:32,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:33,335 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:36,324 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:30:36,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:36,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:36,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:38,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:30:38,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:30:38,263 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:30:39,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:39,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:39,336 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:41,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:41,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:42,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:44,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:44,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:45,338 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:46,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:46,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:47,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:49,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:49,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:49,340 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:49,941 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:30:51,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:51,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:52,943 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:53,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:30:53,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:30:53,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:30:53,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:53,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:55,342 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:30:55,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:55,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:58,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:30:58,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:58,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:30:59,343 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:00,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:00,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:03,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:03,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:03,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:05,345 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:05,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:05,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:08,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:31:08,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:31:08,263 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:31:08,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:08,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:09,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:10,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:10,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:11,347 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:14,060 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:31:14,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:14,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:14,830 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:16,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:16,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:17,349 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:18,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:18,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:19,831 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:19,942 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:31:21,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:21,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:23,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:31:23,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:31:23,264 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:31:23,350 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:23,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:23,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:25,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:26,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:26,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:27,352 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:28,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:28,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:30,734 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:31,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:31,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:33,353 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:33,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:33,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:36,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:36,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:36,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:38,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:31:38,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:31:38,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:31:38,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:38,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:39,355 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:41,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:41,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:41,470 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:43,356 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:43,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:43,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:46,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:46,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:46,778 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:48,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:48,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:49,358 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:49,943 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:31:51,783 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:31:51,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:51,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:51,945 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:53,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:31:53,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:31:53,264 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:31:54,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:54,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:55,361 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:56,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:56,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:57,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:31:59,362 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:31:59,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:31:59,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:01,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:01,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:03,218 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:04,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:04,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:05,363 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:06,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:06,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:08,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:32:08,236 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:32:08,264 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:32:08,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:09,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:09,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:11,365 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:11,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:11,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:13,877 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:14,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:14,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:14,815 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:32:14,817 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:32:14,817 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:32:14,818 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:32:15,366 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:32:16,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:16,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:17,367 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:19,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:19,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:19,819 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:19,946 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:32:21,368 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:21,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:21,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:23,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:32:23,237 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:32:23,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:32:24,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:24,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:25,385 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:26,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:26,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:27,370 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:29,896 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:32:29,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:29,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:30,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:32,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:32,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:33,371 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:34,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:34,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:36,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:37,373 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:37,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:37,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:38,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:32:38,237 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:32:38,264 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:32:39,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:39,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:41,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:42,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:42,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:43,375 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:44,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:44,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:46,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:46,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:47,027 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:49,376 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:49,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:49,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:49,947 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:32:51,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:51,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:52,816 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:53,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:32:53,237 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:32:53,264 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:32:54,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:54,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:55,378 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:56,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:56,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:58,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:32:59,381 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:32:59,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:32:59,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:01,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:01,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:03,671 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:04,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:04,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:05,383 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:07,936 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:33:07,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:07,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:08,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:33:08,237 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:33:08,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:33:09,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:10,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:11,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:11,384 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:13,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:13,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:14,577 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:16,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:16,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:17,386 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:18,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:18,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:19,949 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:33:19,952 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:21,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:21,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:21,387 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:23,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:23,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:23,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:33:23,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:33:23,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:33:25,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:25,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:25,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:27,388 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:27,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:27,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:30,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:30,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:30,943 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:32,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:32,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:33,390 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:35,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:35,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:35,944 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:37,392 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:37,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:37,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:38,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:33:38,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:33:38,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:33:40,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:40,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:41,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:42,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:42,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:43,393 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:45,989 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:33:46,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:46,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:47,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:48,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:48,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:49,395 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:49,951 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:33:50,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:50,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:52,952 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:53,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:33:53,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:33:53,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:33:53,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:53,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:55,397 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:33:55,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:55,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:58,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:33:58,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:58,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:33:59,399 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:00,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:00,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:02,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:02,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:03,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:05,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:05,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:05,401 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:07,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:07,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:08,205 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:08,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:34:08,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:34:08,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:34:09,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:09,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:11,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:12,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:12,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:13,805 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:14,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:14,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:15,404 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:16,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:16,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:18,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:18,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:19,360 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:19,952 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:34:21,406 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:22,270 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:34:22,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:22,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:23,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:34:23,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:34:23,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:34:24,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:25,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:25,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:27,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:27,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:27,408 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:30,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:30,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:30,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:31,022 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:34:31,023 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:34:31,023 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:34:31,024 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:34:31,410 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:34:32,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:32,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:33,410 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:35,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:35,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:36,025 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:37,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:37,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:37,412 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:38,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:34:38,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:34:38,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:34:40,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:40,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:41,358 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:42,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:42,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:43,414 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:45,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:45,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:46,598 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:47,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:47,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:49,416 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:49,954 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:34:50,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:50,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:51,955 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:52,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:52,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:53,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:34:53,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:34:53,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:34:53,417 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:34:55,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:55,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:57,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:57,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:34:57,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:34:59,419 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:00,430 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:35:00,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:00,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:03,098 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:03,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:03,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:05,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:05,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:05,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:08,205 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:08,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:08,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:08,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:35:08,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:35:08,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:35:09,423 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:10,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:10,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:13,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:13,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:13,636 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:15,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:15,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:15,425 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:18,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:18,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:19,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:19,956 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:35:20,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:20,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:21,427 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:22,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:22,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:23,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:35:23,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:35:23,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:35:24,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:24,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:25,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:27,429 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:27,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:27,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:29,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:29,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:30,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:32,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:32,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:33,431 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:34,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:34,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:35,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:37,433 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:38,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:35:38,239 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:35:38,281 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:35:38,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:35:38,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:38,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:41,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:41,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:41,371 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:43,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:43,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:43,435 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:46,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:46,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:46,622 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:48,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:48,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:49,437 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:49,958 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:35:50,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:50,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:51,960 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:52,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:52,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:53,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:35:53,239 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:35:53,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:35:53,438 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:35:55,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:55,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:57,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:35:57,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:57,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:35:59,440 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:00,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:00,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:02,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:02,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:02,946 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:03,442 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:05,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:05,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:07,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:07,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:07,946 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:08,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:36:08,239 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:36:08,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:36:09,444 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:10,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:10,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:12,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:12,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:13,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:15,687 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:36:15,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:15,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:16,446 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:18,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:18,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:19,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:19,959 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:36:20,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:20,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:22,449 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:23,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:36:23,239 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:36:23,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:36:23,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:23,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:24,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:25,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:25,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:26,450 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:28,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:28,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:29,986 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:30,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:30,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:32,452 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:33,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:33,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:35,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:35,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:35,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:38,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:38,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:38,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:36:38,239 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:36:38,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:36:38,454 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:40,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:40,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:40,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:42,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:42,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:44,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:45,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:45,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:46,048 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:46,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:36:46,121 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:36:46,121 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:36:46,122 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:36:46,457 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:36:47,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:47,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:47,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:48,457 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:49,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:49,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:49,960 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:36:51,920 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:53,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:36:53,239 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:36:53,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:36:53,292 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:36:53,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:53,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:54,459 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:36:56,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:56,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:57,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:36:58,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:36:58,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:00,461 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:00,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:00,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:02,643 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:02,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:02,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:04,462 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:05,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:05,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:07,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:07,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:07,793 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:08,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:37:08,239 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:37:08,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:37:10,463 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:10,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:10,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:12,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:12,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:13,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:15,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:15,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:16,465 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:17,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:17,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:18,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:19,962 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:37:20,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:20,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:22,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:22,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:22,467 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:23,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:37:23,240 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:37:23,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:37:24,442 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:25,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:25,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:26,468 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:27,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:27,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:29,752 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:30,537 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:37:30,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:30,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:32,470 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:33,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:33,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:34,753 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:35,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:35,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:38,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:38,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:38,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:37:38,240 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:37:38,266 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:37:38,472 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:40,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:40,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:40,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:42,474 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:43,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:43,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:45,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:45,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:45,570 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:48,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:48,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:48,476 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:49,964 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:37:50,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:50,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:50,965 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:52,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:53,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:53,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:37:53,240 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:37:53,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:37:54,477 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:55,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:55,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:56,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:37:57,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:57,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:58,478 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:37:59,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:37:59,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:01,847 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:02,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:02,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:04,480 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:04,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:04,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:07,701 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:08,168 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:38:08,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:08,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:08,240 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:38:08,240 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:38:08,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:38:10,482 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:10,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:10,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:13,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:13,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:13,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:15,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:15,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:16,484 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:17,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:17,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:18,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:19,966 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:38:20,485 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:20,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:20,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:22,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:22,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:23,207 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:23,240 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:38:23,240 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:38:23,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:38:25,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:25,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:26,487 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:27,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:27,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:28,812 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:30,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:30,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:32,489 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:32,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:32,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:34,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:35,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:35,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:37,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:37,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:38,240 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:38:38,240 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:38:38,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:38:38,491 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:40,364 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:40,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:40,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:42,493 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:42,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:42,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:45,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:45,988 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:38:46,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:46,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:48,494 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:48,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:48,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:49,967 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:38:50,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:50,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:50,968 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:53,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:38:53,241 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:38:53,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:38:53,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:53,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:54,496 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:55,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:55,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:56,386 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:38:58,498 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:38:58,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:38:58,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:00,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:00,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:02,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:02,696 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:39:02,698 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:39:02,698 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:39:02,698 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:39:03,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:03,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:03,499 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:39:03,499 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:04,500 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:05,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:05,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:07,699 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:08,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:39:08,241 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:39:08,268 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:39:08,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:08,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:10,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:10,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:10,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:13,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:13,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:13,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:15,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:15,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:16,504 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:18,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:18,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:18,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:19,968 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:39:20,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:20,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:20,505 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:23,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:39:23,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:39:23,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:39:23,587 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:39:23,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:23,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:24,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:26,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:26,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:26,506 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:28,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:28,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:29,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:31,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:31,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:32,509 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:33,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:33,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:34,985 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:36,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:36,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:36,510 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:38,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:38,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:39:38,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:39:38,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:38,268 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:39:40,375 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:41,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:41,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:42,512 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:43,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:43,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:45,881 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:45,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:45,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:47,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:47,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:48,514 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:49,970 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:39:50,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:50,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:50,971 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:52,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:52,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:53,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:39:53,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:39:53,268 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:39:54,516 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:39:55,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:55,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:56,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:39:57,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:57,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:39:58,517 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:01,080 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:40:01,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:01,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:02,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:03,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:03,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:04,519 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:06,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:06,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:07,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:08,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:40:08,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:40:08,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:40:08,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:08,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:10,520 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:10,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:10,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:12,420 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:13,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:13,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:14,523 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:15,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:15,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:17,807 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:18,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:18,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:19,972 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:40:20,524 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:20,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:20,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:22,973 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:23,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:40:23,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:40:23,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:40:23,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:23,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:25,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:25,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:26,527 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:28,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:28,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:28,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:30,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:30,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:32,528 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:33,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:33,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:34,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:35,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:35,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:36,530 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:38,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:40:38,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:40:38,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:40:38,891 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:40:38,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:38,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:39,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:41,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:41,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:42,532 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:43,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:43,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:45,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:46,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:46,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:48,534 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:48,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:48,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:49,973 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:40:50,975 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:51,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:51,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:53,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:40:53,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:40:53,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:40:53,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:53,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:54,537 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:56,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:40:56,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:56,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:58,538 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:40:58,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:40:58,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:01,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:01,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:01,650 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:03,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:03,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:04,540 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:05,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:05,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:07,501 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:08,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:41:08,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:41:08,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:41:08,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:08,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:10,542 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:10,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:10,689 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:12,824 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:13,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:13,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:14,543 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:17,099 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:41:17,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:17,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:17,825 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:18,733 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:41:18,734 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:41:18,735 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:41:18,735 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:41:19,546 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:41:19,546 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:19,974 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:41:20,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:20,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:20,546 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:22,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:22,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:22,976 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:23,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:41:23,242 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:41:23,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:41:25,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:25,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:26,548 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:27,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:27,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:28,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:29,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:30,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:30,550 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:32,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:32,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:34,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:34,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:34,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:36,551 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:36,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:36,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:38,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:41:38,243 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:41:38,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:41:39,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:39,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:39,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:41,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:41,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:42,554 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:44,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:44,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:45,038 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:46,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:46,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:48,557 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:49,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:49,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:49,976 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:41:50,702 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:51,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:51,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:52,558 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:41:53,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:41:53,243 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:41:53,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:41:55,202 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:41:55,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:55,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:56,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:41:57,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:57,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:41:58,560 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:00,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:00,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:01,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:02,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:02,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:04,562 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:04,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:04,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:07,317 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:07,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:07,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:08,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:42:08,243 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:42:08,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:42:09,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:09,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:10,563 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:12,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:12,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:12,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:14,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:14,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:16,565 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:17,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:17,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:17,911 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:19,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:19,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:19,978 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:42:20,567 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:22,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:22,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:22,981 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:23,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:42:23,243 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:42:23,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:42:24,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:24,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:26,569 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:27,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:27,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:28,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:29,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:29,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:32,393 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:42:32,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:32,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:32,571 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:33,683 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:35,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:35,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:36,572 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:37,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:37,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:38,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:42:38,243 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:42:38,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:42:39,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:40,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:40,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:42,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:42,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:42,574 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:44,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:44,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:45,086 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:46,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:46,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:48,576 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:49,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:49,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:49,979 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:42:50,980 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:51,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:51,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:53,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:42:53,243 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:42:53,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:42:54,578 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:54,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:54,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:56,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:42:56,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:56,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:58,580 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:42:59,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:42:59,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:01,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:01,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:02,014 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:04,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:04,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:04,581 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:06,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:06,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:07,598 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:08,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:43:08,244 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:43:08,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:43:09,663 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:43:09,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:09,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:10,583 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:12,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:12,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:13,337 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:14,584 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:14,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:14,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:17,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:17,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:18,338 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:19,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:19,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:19,981 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:43:20,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:22,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:22,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:23,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:43:23,244 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:43:23,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:43:23,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:24,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:24,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:26,588 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:27,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:27,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:29,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:29,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:29,334 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:30,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:32,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:32,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:34,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:34,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:34,387 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:35,309 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:43:35,310 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:43:35,311 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:43:35,311 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:43:35,591 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:43:36,592 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:36,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:36,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:37,593 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:38,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:43:38,244 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:43:38,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:43:38,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:38,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:39,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:41,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:41,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:42,594 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:43,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:43,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:45,109 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:47,031 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:43:47,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:47,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:48,596 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:49,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:49,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:49,982 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:43:50,984 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:51,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:51,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:52,597 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:53,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:43:53,244 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:43:53,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:43:54,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:54,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:56,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:43:56,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:56,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:58,602 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:43:59,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:43:59,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:01,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:01,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:01,666 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:04,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:04,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:04,604 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:06,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:06,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:07,056 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:08,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:44:08,244 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:44:08,271 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:44:08,606 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:09,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:09,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:11,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:11,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:12,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:14,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:14,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:14,607 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:16,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:16,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:17,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:19,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:19,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:19,985 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:44:20,610 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:21,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:21,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:22,986 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:23,244 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:44:23,245 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:44:23,271 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:44:24,915 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:44:24,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:24,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:26,611 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:27,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:27,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:28,221 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:29,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:29,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:32,613 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:32,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:32,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:34,069 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:34,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:34,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:36,614 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:37,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:37,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:38,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:44:38,245 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:44:38,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:44:39,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:39,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:39,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:41,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:41,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:42,616 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:44,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:44,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:44,777 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:46,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:46,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:48,618 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:49,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:49,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:49,987 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:44:49,989 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:51,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:51,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:52,619 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:53,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:44:53,245 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:44:53,272 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:44:54,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:54,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:55,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:44:56,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:56,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:58,621 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:44:59,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:44:59,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:01,012 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:02,939 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:45:02,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:02,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:04,623 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:05,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:05,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:06,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:07,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:07,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:08,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:45:08,246 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:45:08,272 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:45:10,625 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:10,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:10,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:11,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:12,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:12,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:14,626 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:15,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:15,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:16,563 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:17,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:17,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:19,988 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:45:20,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:20,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:20,628 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:21,989 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:22,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:22,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:23,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:45:23,246 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:45:23,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:45:25,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:25,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:26,630 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:27,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:27,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:27,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:30,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:30,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:30,632 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:32,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:32,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:33,424 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:35,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:35,227 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:36,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:37,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:37,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:38,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:45:38,246 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:45:38,272 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:45:39,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:40,500 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:45:40,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:40,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:42,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:43,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:43,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:44,878 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:45,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:45,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:48,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:48,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:48,638 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:49,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:49,990 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:45:50,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:50,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:51,914 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:45:51,916 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:45:51,916 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:45:51,917 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:45:52,640 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:45:53,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:53,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:53,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:45:53,246 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:45:53,272 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:45:53,640 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:54,640 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:45:55,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:55,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:55,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:45:58,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:58,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:45:58,642 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:00,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:00,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:00,655 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:03,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:03,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:04,644 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:05,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:05,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:06,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:07,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:07,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:08,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:46:08,246 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:46:08,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:46:09,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:10,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:10,646 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:12,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:12,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:12,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:14,647 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:14,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:14,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:18,129 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:46:18,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:18,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:18,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:19,991 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:46:20,649 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:20,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:20,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:22,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:22,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:23,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:46:23,246 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:46:23,272 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:46:23,484 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:25,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:25,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:26,651 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:27,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:27,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:29,166 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:30,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:30,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:32,653 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:32,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:32,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:34,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:35,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:35,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:36,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:37,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:37,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:38,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:46:38,247 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:46:38,272 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:46:40,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:40,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:40,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:42,657 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:42,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:42,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:45,626 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:45,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:45,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:47,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:47,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:48,658 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:49,993 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:46:50,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:50,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:50,994 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:52,660 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:52,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:52,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:53,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:46:53,247 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:46:53,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:46:56,195 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:46:56,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:56,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:56,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:46:58,662 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:46:58,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:46:58,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:00,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:00,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:02,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:03,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:03,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:04,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:05,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:05,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:07,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:07,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:07,994 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:08,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:47:08,247 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:47:08,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:47:10,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:10,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:10,665 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:12,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:12,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:13,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:15,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:15,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:16,667 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:17,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:17,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:18,962 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:19,994 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:47:20,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:20,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:20,668 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:22,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:22,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:23,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:47:23,247 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:47:23,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:47:24,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:25,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:25,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:26,670 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:27,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:27,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:29,776 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:29,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:29,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:32,671 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:33,589 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:47:33,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:33,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:35,271 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:36,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:36,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:36,673 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:38,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:47:38,247 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:47:38,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:47:38,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:38,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:40,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:41,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:41,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:42,674 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:43,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:43,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:46,035 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:46,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:46,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:46,675 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:48,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:48,676 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:49,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:49,996 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:47:50,677 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:51,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:51,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:51,997 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:52,677 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:53,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:47:53,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:47:53,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:47:54,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:54,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:54,873 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:47:54,874 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:47:54,874 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:47:54,874 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:47:55,678 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:47:55,679 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:56,679 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:56,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:56,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:57,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:47:58,679 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:47:59,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:47:59,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:00,681 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:01,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:01,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:02,933 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:04,681 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:04,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:04,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:06,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:06,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:06,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:08,213 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:08,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:48:08,247 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:48:08,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:48:09,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:09,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:10,684 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:12,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:13,532 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:48:13,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:13,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:13,747 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:14,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:15,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:15,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:18,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:18,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:18,686 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:19,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:19,998 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:48:20,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:20,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:20,687 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:22,688 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:23,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:48:23,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:48:23,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:48:23,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:23,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:24,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:25,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:25,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:26,689 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:28,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:28,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:28,689 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:29,933 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:30,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:30,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:30,690 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:32,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:32,690 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:33,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:35,382 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:35,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:35,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:38,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:38,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:38,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:48:38,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:48:38,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:48:40,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:40,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:40,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:42,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:42,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:44,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:44,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:45,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:47,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:47,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:48,695 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:49,999 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:48:50,697 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:51,000 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:51,222 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:48:51,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:51,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:52,697 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:53,247 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:48:53,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:48:53,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:48:54,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:54,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:56,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:56,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:56,699 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:48:56,716 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:48:59,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:59,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:48:59,700 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:01,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:01,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:01,700 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:02,095 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:04,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:04,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:05,701 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:06,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:06,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:07,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:07,702 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:08,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:49:08,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:49:08,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:49:08,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:08,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:09,703 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:11,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:11,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:12,858 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:13,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:13,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:13,578 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:49:13,580 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:49:13,580 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:49:13,580 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:49:13,705 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:49:13,705 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:14,705 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:15,706 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:16,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:16,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:17,706 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:18,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:18,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:18,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:20,001 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:49:21,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:21,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:21,707 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:23,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:49:23,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:49:23,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:49:23,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:23,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:23,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:23,708 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:25,709 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:26,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:26,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:28,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:29,710 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:30,085 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:49:30,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:30,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:31,711 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:32,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:32,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:33,711 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:34,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:35,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:35,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:37,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:37,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:37,713 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:38,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:49:38,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:49:38,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:49:39,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:39,713 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:39,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:40,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:41,714 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:42,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:42,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:45,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:45,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:45,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:45,714 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:47,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:47,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:47,715 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:49,716 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:50,003 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:49:50,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:50,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:51,004 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:52,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:52,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:53,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:49:53,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:49:53,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:49:53,717 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:55,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:55,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:55,718 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:49:56,570 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:49:57,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:57,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:49:57,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:00,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:00,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:01,720 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:01,942 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:02,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:02,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:03,721 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:05,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:05,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:07,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:07,722 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:08,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:50:08,248 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:50:08,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:50:09,041 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:50:09,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:09,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:09,724 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:11,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:12,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:12,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:12,703 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:14,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:14,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:15,726 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:17,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:17,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:17,727 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:18,090 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:19,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:19,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:19,727 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:20,004 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:50:20,804 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:50:20,806 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:50:20,806 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:50:20,806 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:50:21,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:50:22,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:22,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:22,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:23,215 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:23,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:50:23,249 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:50:23,274 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:50:23,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:24,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:24,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:25,729 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:27,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:27,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:27,729 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:28,848 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:29,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:29,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:31,731 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:32,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:32,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:33,732 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:34,227 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:34,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:34,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:35,732 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:37,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:37,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:38,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:50:38,249 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:50:38,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:50:39,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:39,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:39,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:39,734 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:41,734 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:42,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:42,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:43,735 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:44,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:44,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:44,976 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:47,736 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:47,892 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:50:47,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:47,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:49,737 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:50,006 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:50:50,008 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:50,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:50,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:51,737 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:52,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:52,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:53,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:50:53,249 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:50:53,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:50:55,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:50:55,739 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:55,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:55,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:57,739 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:50:57,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:57,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:50:59,740 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:01,112 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:01,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:01,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:03,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:03,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:03,741 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:05,741 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:06,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:06,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:06,490 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:07,742 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:08,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:08,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:51:08,249 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:51:08,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:08,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:51:11,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:11,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:11,743 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:11,871 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:13,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:13,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:13,744 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:15,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:16,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:16,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:17,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:18,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:18,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:19,746 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:20,008 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:51:21,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:21,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:21,747 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:22,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:23,249 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:51:23,249 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:51:23,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:51:23,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:23,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:23,748 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:26,854 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:51:26,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:26,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:27,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:28,002 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:28,028 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:51:28,030 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:51:28,030 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:51:28,030 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:51:28,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:51:28,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:28,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:29,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:30,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:31,750 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:31,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:31,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:33,379 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:33,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:33,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:35,751 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:35,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:35,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:37,751 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:38,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:51:38,250 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:51:38,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:51:38,424 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:38,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:38,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:40,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:40,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:41,753 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:43,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:43,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:43,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:43,753 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:45,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:45,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:45,754 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:47,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:47,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:48,828 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:49,755 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:50,009 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:51:50,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:50,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:51,756 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:52,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:52,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:53,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:51:53,250 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:51:53,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:51:53,757 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:54,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:54,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:54,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:57,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:57,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:57,758 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:51:59,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:59,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:51:59,582 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:51:59,760 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:01,760 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:02,805 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:52:02,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:02,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:04,965 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:05,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:05,762 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:05,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:07,762 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:07,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:07,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:08,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:52:08,250 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:52:08,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:52:09,763 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:10,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:10,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:10,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:12,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:12,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:13,764 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:15,751 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:15,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:15,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:15,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:17,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:17,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:17,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:20,011 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:52:20,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:20,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:21,012 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:21,766 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:22,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:22,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:23,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:52:23,250 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:52:23,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:52:23,767 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:24,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:25,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:25,768 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:26,507 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:27,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:27,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:29,769 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:29,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:29,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:31,769 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:31,892 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:31,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:31,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:33,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:34,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:34,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:35,316 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:52:35,318 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:52:35,319 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:52:35,319 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:52:35,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:52:36,771 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:36,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:36,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:37,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:37,771 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:38,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:52:38,250 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:52:38,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:52:39,772 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:40,191 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:52:40,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:40,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:41,772 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:42,667 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:43,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:43,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:45,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:45,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:45,773 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:47,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:47,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:47,774 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:48,056 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:49,774 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:50,013 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:52:50,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:50,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:52,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:52,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:53,217 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:53,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:52:53,250 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:52:53,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:52:53,776 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:55,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:55,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:55,777 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:52:57,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:57,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:52:58,830 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:52:59,778 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:00,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:00,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:01,778 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:02,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:02,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:03,779 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:04,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:05,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:05,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:07,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:07,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:07,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:08,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:53:08,251 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:53:08,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:53:09,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:09,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:10,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:10,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:11,782 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:12,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:12,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:14,995 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:15,784 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:15,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:15,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:17,788 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:19,473 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:53:19,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:19,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:19,788 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:20,014 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:53:20,016 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:21,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:21,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:23,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:53:23,251 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:53:23,276 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:53:23,790 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:24,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:24,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:25,761 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:25,790 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:26,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:26,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:27,791 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:29,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:29,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:31,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:31,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:31,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:31,792 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:33,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:33,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:33,793 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:35,793 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:36,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:36,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:36,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:38,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:53:38,251 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:53:38,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:53:38,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:38,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:39,794 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:41,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:41,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:41,795 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:41,910 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:42,622 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:53:42,624 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:53:42,624 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:53:42,624 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:53:42,796 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:53:43,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:43,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:43,796 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:44,797 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:46,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:46,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:47,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:47,797 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:48,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:48,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:49,798 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:50,017 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:53:50,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:50,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:51,799 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:52,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:53,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:53:53,251 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:53:53,276 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:53:53,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:53,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:55,800 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:57,067 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:53:57,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:57,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:57,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:53:58,074 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:53:59,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:59,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:53:59,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:01,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:01,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:03,459 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:03,802 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:03,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:03,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:05,803 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:05,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:05,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:07,803 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:08,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:54:08,251 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:54:08,276 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:54:08,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:08,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:09,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:10,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:10,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:11,804 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:12,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:12,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:13,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:14,909 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:15,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:15,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:15,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:17,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:17,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:19,806 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:20,018 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:54:20,020 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:20,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:20,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:21,807 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:22,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:22,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:23,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:54:23,251 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:54:23,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:54:25,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:25,808 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:25,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:25,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:27,809 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:27,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:27,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:29,809 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:29,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:29,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:31,038 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:33,249 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:54:33,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:33,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:33,811 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:35,811 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:36,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:36,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:36,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:37,812 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:38,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:54:38,252 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:54:38,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:38,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:54:38,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:40,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:40,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:41,803 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:41,813 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:43,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:43,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:43,814 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:45,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:45,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:45,814 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:47,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:48,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:48,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:49,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:49,891 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:54:49,892 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:54:49,892 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:54:49,893 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:54:50,019 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:54:50,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:50,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:50,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:54:50,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:51,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:52,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:53,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:53,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:53,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:54:53,252 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:54:53,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:54:53,817 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:55,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:55,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:57,818 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:54:57,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:54:58,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:58,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:54:59,819 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:00,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:00,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:01,820 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:03,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:03,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:03,318 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:05,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:05,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:05,821 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:07,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:07,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:07,822 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:08,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:55:08,252 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:55:08,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:55:08,474 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:09,822 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:10,649 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:55:10,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:10,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:13,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:13,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:13,824 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:14,069 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:15,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:15,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:15,825 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:17,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:17,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:17,826 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:19,473 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:20,021 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:55:20,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:20,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:21,827 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:22,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:22,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:23,251 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:55:23,253 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:55:23,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:55:23,828 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:24,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:24,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:24,847 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:25,828 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:27,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:27,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:29,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:29,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:29,830 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:30,224 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:31,830 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:32,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:32,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:33,831 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:34,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:34,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:35,616 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:36,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:36,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:37,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:38,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:55:38,252 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:55:38,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:55:39,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:39,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:39,833 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:41,012 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:41,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:41,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:41,834 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:44,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:44,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:45,836 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:46,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:47,758 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:55:47,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:47,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:47,837 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:49,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:49,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:50,023 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:55:51,783 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:51,838 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:52,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:52,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:53,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:55:53,252 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:55:53,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:55:53,838 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:54,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:54,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:55,839 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:57,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:55:57,203 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:55:57,204 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:55:57,205 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:55:57,206 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:55:57,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:57,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:57,840 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:55:58,840 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:55:59,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:59,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:55:59,841 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:01,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:02,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:02,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:02,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:03,844 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:04,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:04,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:06,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:06,637 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:07,845 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:07,930 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:08,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:56:08,253 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:56:08,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:56:08,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:08,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:09,846 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:11,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:11,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:11,847 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:13,311 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:13,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:13,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:15,848 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:16,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:16,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:17,849 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:18,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:18,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:18,686 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:19,849 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:20,024 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:56:21,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:21,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:23,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:56:23,253 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:56:23,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:56:23,851 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:24,079 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:25,255 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:56:25,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:25,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:25,852 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:27,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:27,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:27,852 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:29,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:30,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:30,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:31,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:32,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:32,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:33,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:34,850 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:35,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:35,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:35,855 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:37,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:37,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:38,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:56:38,253 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:56:38,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:56:39,857 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:40,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:40,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:40,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:41,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:42,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:42,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:43,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:45,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:45,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:45,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:47,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:47,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:47,860 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:49,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:49,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:49,861 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:50,026 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:56:51,001 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:51,862 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:52,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:52,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:53,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:56:53,253 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:56:53,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:56:54,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:54,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:55,863 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:56,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:56,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:56,381 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:56:57,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:56:58,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:58,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:56:59,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:01,768 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:02,530 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:57:02,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:02,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:03,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:04,494 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:57:04,496 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:57:04,496 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:57:04,496 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:57:04,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:57:05,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:05,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:05,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:06,868 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:07,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:07,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:07,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:07,868 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:08,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:57:08,253 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:57:08,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:57:10,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:10,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:11,869 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:12,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:12,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:12,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:13,870 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:15,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:15,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:17,871 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:17,911 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:17,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:17,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:19,872 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:20,027 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:57:20,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:20,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:21,873 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:22,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:22,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:23,219 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:23,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:57:23,253 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:57:23,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:57:25,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:25,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:25,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:27,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:27,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:27,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:28,660 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:29,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:29,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:29,875 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:32,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:32,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:33,877 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:34,046 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:34,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:34,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:35,877 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:37,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:37,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:37,878 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:38,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:57:38,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:57:38,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:57:39,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:40,837 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:57:40,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:40,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:41,880 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:43,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:43,881 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:43,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:44,796 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:45,881 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:45,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:45,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:49,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:49,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:49,883 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:50,030 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:57:50,032 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:51,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:51,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:51,883 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:53,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:57:53,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:57:53,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:57:53,884 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:54,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:54,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:55,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:57:56,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:56,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:57,886 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:57:59,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:59,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:57:59,886 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:00,921 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:01,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:01,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:01,887 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:04,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:04,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:05,889 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:06,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:06,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:06,300 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:07,889 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:08,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:58:08,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:58:08,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:58:09,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:09,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:09,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:11,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:11,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:11,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:11,708 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:58:11,710 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:58:11,710 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:58:11,710 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:58:11,891 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:58:12,892 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:13,892 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:14,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:14,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:15,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:16,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:16,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:17,070 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:17,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:19,841 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:58:19,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:19,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:20,031 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:58:21,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:22,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:22,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:22,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:23,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:58:23,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:58:23,279 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:58:23,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:24,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:24,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:25,896 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:27,822 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:28,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:28,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:29,897 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:30,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:30,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:31,898 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:32,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:32,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:33,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:33,899 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:35,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:35,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:37,900 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:38,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:38,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:38,219 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:38,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:58:38,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:58:38,279 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:58:39,900 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:40,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:40,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:41,901 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:42,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:42,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:44,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:44,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:44,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:47,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:47,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:49,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:49,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:49,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:50,032 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:58:52,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:52,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:53,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:58:53,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:58:53,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:58:54,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:54,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:54,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:55,905 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:58:57,631 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:58:57,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:57,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:58:59,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:58:59,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:00,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:00,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:01,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:02,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:02,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:03,907 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:05,052 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:05,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:05,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:07,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:07,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:07,908 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:08,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:59:08,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:59:08,279 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:59:09,908 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:10,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:10,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:10,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:11,909 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:12,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:12,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:15,817 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:15,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:15,910 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:15,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:17,910 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:17,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:17,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:19,911 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:20,034 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:59:20,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:20,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:21,035 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:22,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:23,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:23,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:59:23,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:59:23,279 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:59:23,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:25,913 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:26,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:26,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:26,556 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:27,913 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:28,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:28,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:29,964 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 23:59:29,965 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-27 23:59:29,966 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-27 23:59:29,966 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 23:59:30,914 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-27 23:59:30,914 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:31,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:31,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:31,914 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:31,967 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:33,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:33,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:33,915 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:35,915 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:36,472 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 23:59:36,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:36,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:37,303 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:38,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:59:38,254 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:59:38,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:59:39,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:39,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:39,916 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:41,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:41,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:41,916 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:42,670 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:43,917 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:44,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:44,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:46,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:46,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:47,918 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:48,038 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:49,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:49,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:49,919 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:50,035 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-27 23:59:51,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:51,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:53,221 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:53,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 23:59:53,255 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-27 23:59:53,279 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 23:59:53,920 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:54,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:54,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:55,920 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:56,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:56,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:57,921 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-27 23:59:58,781 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 23:59:59,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 23:59:59,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:01,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:01,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:01,922 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:03,923 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:04,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:04,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:04,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:05,923 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:06,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:06,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:08,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:00:08,255 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:00:08,280 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:00:09,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:09,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:09,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:09,924 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:11,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:11,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:11,924 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:13,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:14,912 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:14,995 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:00:15,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:15,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:17,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:17,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:17,926 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:19,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:19,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:19,926 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:20,037 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:00:20,039 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:21,927 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:22,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:22,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:23,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:00:23,255 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:00:23,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:00:24,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:24,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:25,650 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:25,928 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:27,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:27,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:27,928 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:29,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:29,929 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:29,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:31,021 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:31,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:31,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:33,929 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:34,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:34,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:35,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:36,387 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:36,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:36,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:37,098 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:00:37,099 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:00:37,100 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:00:37,100 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:00:37,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:00:37,931 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:38,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:00:38,255 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:00:38,280 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:00:38,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:39,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:39,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:41,755 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:41,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:41,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:41,931 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:43,932 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:44,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:44,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:45,932 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:46,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:46,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:47,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:49,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:49,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:49,934 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:50,038 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:00:51,934 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:52,496 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:53,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:00:53,255 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:00:53,280 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:00:53,453 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:00:53,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:53,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:53,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:56,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:56,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:57,871 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:00:57,936 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:00:58,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:58,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:00:59,937 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:01,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:01,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:01,938 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:03,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:03,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:03,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:05,939 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:06,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:06,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:07,939 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:08,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:01:08,255 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:01:08,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:01:08,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:08,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:08,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:09,940 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:11,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:11,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:13,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:13,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:13,941 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:14,010 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:15,942 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:16,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:16,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:17,942 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:18,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:18,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:19,376 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:20,039 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:01:21,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:21,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:21,943 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:23,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:01:23,255 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:01:23,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:01:23,944 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:23,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:23,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:24,383 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:25,945 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:26,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:26,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:28,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:28,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:29,946 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:30,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:31,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:32,388 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:01:32,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:32,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:33,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:35,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:35,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:35,495 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:37,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:37,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:37,948 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:38,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:01:38,255 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:01:38,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:01:39,949 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:40,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:40,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:40,878 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:42,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:42,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:43,950 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:44,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:01:44,531 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:01:44,531 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:01:44,531 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:01:44,951 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:01:45,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:45,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:45,951 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:46,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:46,951 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:47,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:47,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:47,952 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:50,042 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:01:50,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:50,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:51,953 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:52,043 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:52,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:52,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:53,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:01:53,256 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:01:53,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:01:53,954 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:55,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:55,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:55,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:01:57,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:01:57,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:57,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:01:59,956 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:00,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:00,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:01,957 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:02,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:02,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:02,605 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:03,957 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:04,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:04,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:07,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:07,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:07,958 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:07,973 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:08,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:02:08,256 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:02:08,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:02:09,959 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:10,994 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:02:11,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:11,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:11,959 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:13,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:13,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:13,347 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:15,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:15,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:15,960 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:17,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:17,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:17,961 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:18,718 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:19,962 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:20,044 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:02:21,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:21,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:23,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:23,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:23,255 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:02:23,256 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:02:23,281 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:02:23,963 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:24,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:25,964 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:26,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:26,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:27,965 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:28,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:28,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:29,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:30,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:30,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:31,966 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:32,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:32,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:33,967 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:34,823 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:34,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:34,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:35,967 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:38,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:38,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:38,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:02:38,256 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:02:38,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:02:39,968 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:40,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:40,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:40,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:41,970 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:43,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:43,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:43,970 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:45,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:45,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:45,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:47,971 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:48,426 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:02:48,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:48,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:49,972 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:50,045 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:02:51,046 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:51,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:51,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:51,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:02:51,640 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:02:51,640 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:02:51,641 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:02:51,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:02:51,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:52,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:53,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:02:53,256 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:02:53,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:02:53,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:53,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:55,975 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:56,294 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:02:56,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:56,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:57,975 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:02:58,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:02:58,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:01,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:01,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:01,662 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:01,977 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:03,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:03,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:03,977 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:05,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:06,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:06,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:07,035 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:08,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:03:08,256 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:03:08,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:03:08,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:08,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:09,980 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:10,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:10,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:11,980 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:12,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:13,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:13,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:13,981 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:15,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:15,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:17,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:17,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:17,782 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:17,982 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:19,983 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:20,046 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:03:20,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:20,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:21,984 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:22,172 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:22,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:23,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:23,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:03:23,256 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:03:23,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:03:25,710 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:03:25,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:25,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:25,985 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:27,985 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:28,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:28,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:28,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:29,986 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:30,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:30,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:33,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:33,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:33,916 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:33,987 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:35,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:35,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:35,988 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:37,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:37,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:37,989 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:38,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:03:38,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:03:38,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:03:39,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:40,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:40,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:41,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:42,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:42,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:43,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:44,649 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:45,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:45,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:46,991 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:47,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:47,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:50,004 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:50,048 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:03:50,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:50,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:50,992 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:52,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:52,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:52,993 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:53,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:03:53,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:03:53,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:03:54,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:03:55,464 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:03:55,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:55,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:57,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:57,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:03:58,767 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:03:58,768 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:03:58,769 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:03:58,769 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:03:58,995 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:03:58,996 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:00,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:00,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:00,771 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:00,996 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:02,890 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:04:02,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:02,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:05,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:05,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:05,771 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:07,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:07,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:08,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:04:08,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:04:08,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:04:09,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:09,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:11,170 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:12,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:12,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:12,999 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:14,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:14,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:16,758 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:16,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:16,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:19,001 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:19,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:19,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:20,050 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:04:21,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:21,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:22,374 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:23,002 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:23,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:04:23,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:04:23,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:04:24,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:24,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:26,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:26,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:28,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:29,003 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:29,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:29,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:31,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:31,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:33,610 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:34,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:34,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:35,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:36,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:36,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:38,256 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:04:38,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:04:38,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:04:39,372 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:04:39,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:39,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:39,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:41,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:42,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:42,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:44,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:44,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:44,879 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:47,008 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:47,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:47,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:49,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:49,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:50,051 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:04:50,054 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:51,009 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:51,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:51,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:53,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:04:53,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:04:53,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:04:53,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:53,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:55,081 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:04:56,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:56,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:57,011 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:04:58,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:04:58,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:00,085 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:01,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:01,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:03,012 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:03,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:03,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:05,699 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:06,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:06,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:08,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:05:08,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:05:08,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:05:08,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:08,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:09,014 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:11,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:11,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:11,493 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:13,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:13,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:15,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:16,630 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:05:16,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:16,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:16,978 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:19,017 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:19,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:19,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:20,053 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:05:21,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:21,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:22,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:23,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:05:23,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:05:23,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:05:24,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:24,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:25,019 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:26,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:26,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:27,429 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:29,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:29,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:31,020 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:31,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:31,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:32,721 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:34,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:34,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:36,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:36,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:37,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:38,223 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:38,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:05:38,257 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:05:38,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:05:39,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:39,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:41,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:41,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:43,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:43,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:43,941 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:43,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:45,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:45,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:47,025 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:48,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:48,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:49,556 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:50,055 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:05:50,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:50,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:53,027 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:53,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:05:53,258 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:05:53,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:05:54,183 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:05:54,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:54,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:55,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:05:57,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:57,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:59,028 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:05:59,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:05:59,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:00,806 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:02,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:02,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:04,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:04,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:05,030 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:06,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:06,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:06,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:08,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:06:08,258 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:06:08,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:06:08,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:08,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:11,032 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:11,486 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:11,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:11,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:13,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:13,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:15,034 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:16,628 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:16,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:16,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:18,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:18,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:20,057 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:06:21,035 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:21,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:21,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:22,058 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:23,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:06:23,258 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:06:23,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:06:23,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:23,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:26,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:26,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:27,037 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:27,834 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:28,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:28,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:32,005 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:06:32,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:32,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:33,039 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:33,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:34,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:34,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:36,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:36,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:38,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:06:38,258 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:06:38,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:06:38,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:39,041 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:39,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:39,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:41,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:41,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:43,042 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:43,782 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:44,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:44,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:46,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:46,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:49,043 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:49,385 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:49,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:49,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:50,058 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:06:51,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:51,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:53,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:06:53,258 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:06:53,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:06:54,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:54,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:54,460 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:06:55,045 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:06:56,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:56,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:59,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:59,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:06:59,701 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:01,047 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:01,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:01,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:04,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:04,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:05,276 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:06,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:06,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:07,048 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:08,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:07:08,258 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:07:08,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:07:09,403 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:07:09,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:09,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:10,860 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:11,050 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:12,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:12,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:14,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:14,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:16,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:16,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:16,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:17,051 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:18,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:18,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:20,060 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:07:21,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:21,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:21,938 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:23,053 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:23,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:07:23,258 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:07:23,284 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:07:23,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:23,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:26,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:26,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:27,618 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:28,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:28,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:29,054 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:31,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:31,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:33,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:33,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:33,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:35,056 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:36,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:36,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:38,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:07:38,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:07:38,284 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:07:38,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:38,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:38,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:39,057 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:41,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:41,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:43,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:43,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:43,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:45,059 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:46,889 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:07:46,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:46,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:48,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:49,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:49,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:50,061 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:07:51,064 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:51,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:51,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:53,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:07:53,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:07:53,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:07:54,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:07:54,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:54,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:56,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:56,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:57,066 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:07:59,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:59,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:07:59,848 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:01,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:01,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:03,067 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:04,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:04,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:05,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:06,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:06,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:07,069 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:08,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:08:08,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:08:08,284 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:08:09,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:09,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:10,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:11,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:11,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:13,070 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:14,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:14,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:15,702 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:16,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:16,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:19,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:19,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:19,072 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:20,063 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:08:21,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:21,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:21,064 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:23,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:08:23,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:08:23,284 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:08:24,239 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:08:24,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:24,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:25,074 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:26,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:27,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:27,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:29,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:29,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:31,076 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:32,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:32,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:32,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:34,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:34,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:37,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:37,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:37,083 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:38,140 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:38,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:08:38,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:08:38,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:08:39,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:39,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:41,084 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:41,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:41,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:43,728 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:43,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:43,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:46,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:46,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:47,086 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:48,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:48,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:49,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:50,064 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:08:51,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:51,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:53,087 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:53,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:08:53,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:08:53,284 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:08:53,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:53,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:54,476 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:08:55,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:55,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:58,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:58,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:08:59,089 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:08:59,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:01,958 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:09:01,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:01,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:04,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:04,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:05,090 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:05,563 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:07,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:07,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:08,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:09:08,260 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:09:08,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:09:09,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:09,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:11,092 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:11,426 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:11,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:11,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:14,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:14,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:15,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:16,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:16,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:16,829 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:19,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:19,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:20,066 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:09:21,095 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:21,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:21,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:22,067 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:23,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:09:23,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:09:23,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:09:24,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:24,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:26,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:26,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:27,097 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:28,027 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:29,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:29,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:31,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:31,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:33,099 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:33,644 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:34,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:34,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:36,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:36,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:38,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:09:38,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:09:38,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:09:39,101 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:39,405 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:09:39,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:39,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:39,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:42,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:42,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:43,102 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:44,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:44,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:44,890 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:47,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:47,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:49,104 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:49,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:49,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:50,067 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:09:50,069 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:52,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:52,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:53,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:09:53,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:09:53,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:09:54,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:54,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:55,106 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:09:55,418 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:09:57,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:57,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:59,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:09:59,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:00,601 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:01,108 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:01,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:01,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:03,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:03,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:06,327 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:06,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:06,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:07,110 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:08,259 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:10:08,259 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:10:08,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:10:08,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:08,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:11,112 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:11,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:11,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:11,973 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:13,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:13,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:17,114 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:17,170 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:10:17,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:17,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:17,568 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:19,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:19,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:20,069 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:10:21,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:21,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:23,115 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:23,146 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:23,260 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:10:23,261 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:10:23,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:10:24,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:24,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:26,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:26,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:28,799 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:29,117 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:29,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:29,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:31,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:31,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:34,371 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:34,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:34,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:35,119 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:36,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:36,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:38,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:10:38,261 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:10:38,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:10:39,120 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:39,488 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:39,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:39,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:41,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:41,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:44,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:44,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:44,639 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:45,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:46,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:46,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:49,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:49,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:50,070 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:10:50,071 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:51,124 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:51,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:51,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:53,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:10:53,261 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:10:53,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:10:54,742 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:10:54,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:54,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:55,857 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:10:57,126 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:10:57,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:57,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:59,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:10:59,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:01,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:02,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:02,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:03,128 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:04,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:04,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:07,062 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:07,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:07,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:07,130 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:08,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:11:08,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:11:08,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:11:09,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:09,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:12,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:12,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:12,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:13,132 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:14,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:14,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:16,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:16,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:18,204 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:18,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:18,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:19,133 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:20,072 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:11:21,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:21,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:23,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:23,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:11:23,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:11:23,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:11:23,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:23,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:25,135 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:25,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:25,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:28,343 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:28,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:28,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:31,137 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:32,018 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:11:32,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:32,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:33,346 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:34,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:34,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:35,138 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:36,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:36,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:38,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:11:38,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:11:38,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:11:38,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:39,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:39,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:41,140 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:41,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:41,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:43,502 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:44,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:44,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:46,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:46,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:47,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:49,081 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:49,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:49,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:50,073 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:11:51,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:51,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:53,144 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:53,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:11:53,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:11:53,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:11:54,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:11:54,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:54,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:56,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:56,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:59,147 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:11:59,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:11:59,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:00,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:01,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:01,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:03,148 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:04,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:04,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:06,105 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:06,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:06,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:08,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:12:08,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:12:08,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:12:09,150 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:09,904 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:12:09,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:09,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:11,488 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:12,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:12,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:14,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:14,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:15,152 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:17,203 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:17,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:17,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:19,679 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:19,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:20,075 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:12:21,154 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:22,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:22,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:23,076 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:23,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:12:23,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:12:23,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:12:24,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:24,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:27,156 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:27,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:27,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:28,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:29,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:29,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:31,158 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:32,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:32,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:33,995 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:34,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:34,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:36,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:36,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:37,159 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:38,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:12:38,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:12:38,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:12:39,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:39,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:39,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:41,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:41,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:43,162 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:44,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:44,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:45,294 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:47,549 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:12:47,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:47,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:49,164 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:50,077 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:12:50,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:50,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:51,079 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:52,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:52,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:53,165 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:12:53,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:12:53,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:12:53,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:12:55,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:55,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:56,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:12:57,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:57,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:12:59,167 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:00,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:00,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:02,057 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:02,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:02,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:05,168 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:05,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:05,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:07,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:07,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:07,652 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:08,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:13:08,262 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:13:08,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:13:10,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:10,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:11,171 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:12,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:12,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:13,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:15,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:15,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:17,172 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:17,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:17,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:18,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:19,032 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:13:19,034 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:13:19,035 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:13:19,035 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:13:19,173 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:13:19,174 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:20,078 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:13:20,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:20,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:20,174 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:21,174 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:22,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:23,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:24,080 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:26,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:13:26,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:13:26,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:13:26,731 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:13:26,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:26,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:29,176 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:29,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:29,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:29,748 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:31,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:31,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:33,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:33,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:34,749 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:36,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:36,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:38,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:38,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:39,750 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:41,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:41,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:41,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:13:41,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:13:41,518 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:13:43,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:43,302 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:45,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:45,740 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:45,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:47,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:47,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:49,182 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:50,079 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:13:50,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:50,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:51,081 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:51,183 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:52,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:52,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:53,184 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:55,185 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:13:55,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:55,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:56,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:13:56,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:13:56,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:13:56,518 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:13:57,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:57,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:13:59,186 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:00,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:00,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:01,187 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:01,837 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:03,188 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:04,215 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:14:04,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:04,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:07,184 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:07,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:07,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:07,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:09,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:09,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:09,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:11,191 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:11,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:14:11,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:14:11,518 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:14:12,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:12,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:12,659 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:14,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:14,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:15,192 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:17,193 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:17,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:17,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:17,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:19,194 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:19,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:19,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:20,082 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:14:22,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:22,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:23,195 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:23,248 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:24,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:24,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:25,196 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:26,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:14:26,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:14:26,518 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:14:27,197 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:27,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:27,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:28,673 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:29,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:29,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:31,198 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:32,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:32,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:33,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:33,964 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:34,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:34,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:35,200 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:36,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:36,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:39,201 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:39,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:39,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:39,325 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:41,202 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:41,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:14:41,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:14:41,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:14:42,808 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:14:42,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:42,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:43,202 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:44,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:44,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:44,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:47,205 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:47,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:47,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:49,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:49,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:50,026 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:50,084 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:14:50,206 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:52,207 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:52,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:52,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:54,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:54,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:55,386 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:14:56,087 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:14:56,089 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:14:56,089 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:14:56,090 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:14:56,208 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:14:56,209 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:56,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:14:56,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:14:56,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:14:57,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:57,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:58,209 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:14:59,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:14:59,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:00,210 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:00,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:02,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:02,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:04,212 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:04,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:04,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:06,112 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:06,212 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:07,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:07,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:08,213 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:09,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:09,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:11,463 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:11,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:15:11,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:15:11,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:15:12,215 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:12,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:12,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:14,216 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:14,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:14,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:16,217 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:16,835 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:17,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:17,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:20,085 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:15:20,219 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:21,504 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:15:21,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:21,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:22,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:22,220 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:23,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:23,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:24,220 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:26,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:26,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:26,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:15:26,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:15:26,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:15:27,732 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:28,222 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:28,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:28,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:30,223 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:31,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:31,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:32,224 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:32,908 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:33,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:33,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:36,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:36,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:36,225 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:38,226 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:38,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:38,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:38,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:40,226 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:40,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:40,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:41,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:15:41,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:15:41,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:15:43,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:43,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:43,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:44,228 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:45,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:45,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:46,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:48,230 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:48,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:48,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:48,997 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:50,087 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:15:50,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:50,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:52,231 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:52,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:52,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:54,232 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:54,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:15:55,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:55,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:56,233 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:15:56,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:15:56,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:15:56,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:15:58,742 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:15:58,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:58,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:15:59,732 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:00,234 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:01,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:01,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:02,235 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:03,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:16:03,121 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:16:03,122 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:16:03,122 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:16:03,235 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:16:03,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:03,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:04,236 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:05,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:05,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:05,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:08,237 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:08,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:08,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:10,238 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:10,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:10,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:10,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:11,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:16:11,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:16:11,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:16:12,238 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:13,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:13,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:15,826 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:15,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:15,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:16,240 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:18,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:18,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:19,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:20,088 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:16:20,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:21,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:21,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:21,090 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:24,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:24,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:24,243 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:26,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:26,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:26,243 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:26,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:16:26,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:16:26,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:16:26,738 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:29,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:29,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:30,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:31,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:31,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:31,932 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:32,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:34,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:34,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:34,246 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:37,302 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:37,591 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:16:37,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:37,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:38,248 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:39,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:39,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:40,249 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:41,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:16:41,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:16:41,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:16:42,250 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:42,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:42,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:42,730 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:44,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:44,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:46,252 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:47,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:47,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:48,026 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:48,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:49,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:49,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:50,090 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:16:50,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:52,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:52,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:53,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:54,255 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:54,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:54,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:56,255 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:56,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:16:56,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:16:56,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:16:57,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:57,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:58,256 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:16:58,760 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:16:59,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:16:59,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:02,258 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:02,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:02,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:04,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:04,259 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:04,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:04,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:06,260 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:07,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:07,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:09,478 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:09,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:09,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:10,186 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:17:10,188 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:17:10,188 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:17:10,189 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:17:10,262 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:17:10,262 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:11,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:17:11,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:17:11,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:17:12,262 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:12,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:12,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:14,263 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:14,850 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:16,589 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:17:16,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:16,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:18,264 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:19,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:19,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:20,091 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:17:20,093 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:20,265 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:21,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:21,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:22,266 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:23,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:23,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:25,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:26,267 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:26,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:17:26,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:17:26,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:17:26,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:26,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:28,268 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:28,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:28,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:30,269 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:30,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:31,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:31,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:33,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:33,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:34,270 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:36,271 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:36,315 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:36,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:36,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:38,272 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:38,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:38,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:41,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:17:41,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:17:41,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:17:41,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:41,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:41,750 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:42,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:43,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:43,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:44,275 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:46,275 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:46,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:46,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:47,056 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:48,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:48,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:50,093 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:17:50,277 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:51,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:51,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:52,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:52,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:54,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:17:55,453 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:17:55,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:55,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:56,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:17:56,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:17:56,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:17:57,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:57,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:17:57,671 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:17:58,280 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:00,280 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:00,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:00,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:02,281 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:02,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:02,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:03,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:05,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:05,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:06,282 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:07,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:07,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:08,283 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:08,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:10,284 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:10,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:10,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:11,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:18:11,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:18:11,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:18:12,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:12,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:13,898 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:14,285 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:15,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:15,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:16,286 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:17,283 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:18:17,285 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:18:17,285 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:18:17,285 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:18:17,288 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:18:17,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:17,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:18,287 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:19,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:20,094 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:18:21,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:21,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:22,288 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:23,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:23,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:24,289 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:24,608 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:26,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:26,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:26,289 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:26,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:18:26,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:18:26,520 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:18:28,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:28,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:29,963 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:30,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:30,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:30,290 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:32,291 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:33,293 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:18:33,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:33,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:34,292 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:35,316 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:36,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:36,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:38,293 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:38,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:38,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:40,294 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:40,424 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:41,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:41,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:41,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:18:41,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:18:41,521 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:18:43,217 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:43,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:45,739 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:45,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:45,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:47,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:47,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:50,096 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:18:50,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:50,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:51,098 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:52,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:52,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:54,298 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:55,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:55,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:56,221 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:18:56,298 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:18:56,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:18:56,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:18:56,521 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:18:57,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:18:57,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:00,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:00,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:00,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:01,579 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:02,300 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:02,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:02,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:04,300 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:05,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:05,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:06,938 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:07,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:07,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:08,302 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:10,303 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:11,472 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:19:11,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:11,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:11,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:19:11,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:19:11,521 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:19:12,303 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:12,711 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:14,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:14,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:16,305 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:16,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:16,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:18,305 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:18,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:19,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:19,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:20,098 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:19:20,306 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:21,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:21,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:24,001 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:24,307 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:24,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:24,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:26,308 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:26,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:26,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:26,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:19:26,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:19:26,521 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:19:28,308 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:29,350 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:29,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:29,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:31,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:31,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:32,310 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:34,311 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:34,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:34,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:34,697 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:35,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:19:35,404 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:19:35,404 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:19:35,405 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:19:36,311 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:19:36,312 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:36,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:36,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:39,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:39,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:40,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:40,312 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:41,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:41,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:41,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:19:41,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:19:41,521 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:19:42,313 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:44,313 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:44,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:44,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:45,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:46,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:46,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:48,315 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:49,948 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:19:49,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:49,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:50,099 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:19:50,315 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:51,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:52,316 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:53,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:53,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:55,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:55,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:56,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:19:56,317 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:19:56,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:19:56,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:19:56,521 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:19:58,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:58,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:19:58,318 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:00,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:00,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:00,318 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:01,498 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:03,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:03,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:04,319 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:05,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:05,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:06,320 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:06,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:08,321 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:08,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:08,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:10,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:10,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:11,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:20:11,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:20:11,521 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:20:12,322 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:12,702 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:13,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:13,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:14,323 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:15,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:15,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:18,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:18,325 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:18,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:18,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:20,101 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:20:20,325 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:20,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:20,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:22,326 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:22,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:22,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:23,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:25,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:25,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:26,327 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:26,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:20:26,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:20:26,522 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:20:28,328 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:28,713 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:28,834 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:20:28,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:28,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:30,329 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:30,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:30,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:33,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:33,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:34,300 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:34,330 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:35,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:35,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:36,330 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:38,331 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:39,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:39,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:39,656 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:41,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:41,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:41,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:20:41,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:20:41,522 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:20:42,332 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:42,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:20:42,363 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:20:42,363 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:20:42,363 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:20:43,333 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:20:44,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:44,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:44,333 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:45,015 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:46,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:46,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:46,333 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:49,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:49,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:50,103 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:20:50,104 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:50,335 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:51,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:51,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:52,335 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:53,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:53,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:54,336 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:20:55,716 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:20:56,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:56,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:56,520 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:20:56,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:20:56,522 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:20:58,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:58,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:20:58,337 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:00,337 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:01,070 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:01,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:01,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:02,338 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:03,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:03,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:06,339 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:06,425 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:06,815 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:21:06,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:06,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:08,340 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:09,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:09,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:10,340 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:11,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:21:11,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:21:11,522 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:21:11,747 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:11,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:11,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:14,341 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:14,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:14,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:16,342 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:16,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:16,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:17,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:18,343 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:19,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:19,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:20,104 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:21:21,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:21,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:22,344 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:22,488 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:24,344 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:24,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:25,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:26,345 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:26,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:21:26,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:21:26,522 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:21:27,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:27,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:27,700 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:29,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:29,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:30,346 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:32,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:32,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:32,347 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:33,210 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:34,347 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:34,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:34,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:36,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:36,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:38,348 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:38,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:39,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:39,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:40,349 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:41,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:21:41,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:21:41,522 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:21:42,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:42,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:42,350 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:43,671 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:45,451 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:21:45,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:45,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:46,351 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:48,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:48,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:48,352 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:49,275 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:49,302 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:21:49,303 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:21:49,303 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:21:49,304 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:21:49,352 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:21:50,106 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:21:50,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:50,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:50,352 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:53,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:53,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:54,354 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:54,625 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:21:55,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:55,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:56,354 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:56,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:21:56,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:21:56,523 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:21:58,355 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:21:58,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:58,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:21:59,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:00,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:00,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:02,356 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:03,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:03,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:04,356 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:05,343 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:05,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:05,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:06,357 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:08,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:08,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:10,358 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:10,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:10,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:10,692 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:11,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:22:11,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:22:11,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:22:12,359 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:13,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:13,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:14,360 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:15,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:15,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:16,060 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:17,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:17,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:18,361 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:20,108 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:22:20,361 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:20,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:20,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:21,109 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:22,362 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:23,965 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:22:23,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:24,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:26,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:26,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:26,364 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:26,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:22:26,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:22:26,523 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:22:26,657 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:28,364 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:28,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:28,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:30,364 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:30,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:30,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:32,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:33,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:33,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:34,366 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:35,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:35,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:36,366 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:37,469 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:38,367 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:38,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:38,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:40,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:40,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:41,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:22:41,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:22:41,523 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:22:42,368 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:42,673 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:43,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:43,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:44,369 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:45,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:45,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:46,369 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:47,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:47,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:48,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:50,109 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:22:50,370 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:51,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:51,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:52,371 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:53,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:53,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:53,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:22:54,372 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:56,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:22:56,231 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:22:56,231 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:22:56,231 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:22:56,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:56,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:56,373 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:22:56,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:22:56,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:22:56,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:22:58,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:58,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:22:58,373 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:22:58,713 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:00,374 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:01,682 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:23:01,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:01,710 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:02,375 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:04,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:04,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:04,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:06,376 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:06,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:06,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:08,377 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:09,579 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:09,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:09,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:10,378 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:11,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:23:11,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:23:11,523 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:23:11,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:11,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:14,379 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:14,645 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:14,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:14,925 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:16,379 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:16,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:16,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:18,380 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:19,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:19,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:20,111 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:23:20,113 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:21,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:21,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:22,381 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:24,382 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:24,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:24,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:25,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:26,383 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:26,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:23:26,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:23:26,523 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:23:26,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:26,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:29,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:29,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:30,384 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:30,996 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:31,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:31,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:32,385 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:34,386 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:34,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:34,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:36,339 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:36,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:36,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:38,387 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:40,336 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:23:40,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:40,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:40,388 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:41,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:23:41,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:23:41,523 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:23:41,749 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:42,389 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:43,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:43,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:45,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:45,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:46,391 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:47,070 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:48,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:48,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:48,391 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:50,112 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:23:50,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:50,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:50,392 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:52,114 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:53,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:53,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:54,393 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:55,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:55,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:56,394 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:23:56,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:23:56,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:23:56,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:23:57,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:23:58,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:58,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:23:58,395 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:00,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:00,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:02,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:03,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:03,164 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:24:03,165 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:24:03,166 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:24:03,166 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:24:03,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:03,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:03,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:24:04,397 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:05,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:05,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:06,397 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:08,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:08,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:08,507 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:10,399 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:10,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:10,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:11,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:24:11,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:24:11,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:24:12,400 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:13,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:13,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:13,712 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:14,400 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:15,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:15,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:18,402 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:18,985 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:24:19,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:19,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:19,216 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:20,114 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:24:20,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:21,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:21,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:22,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:23,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:23,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:24,572 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:26,405 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:26,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:24:26,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:24:26,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:24:26,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:26,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:28,406 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:28,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:28,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:29,926 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:31,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:31,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:32,408 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:33,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:33,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:34,409 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:35,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:36,409 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:36,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:36,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:38,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:38,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:40,411 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:40,642 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:41,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:24:41,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:24:41,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:24:42,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:42,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:42,411 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:44,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:44,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:44,412 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:45,989 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:47,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:47,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:48,413 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:49,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:49,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:50,115 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:24:50,414 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:51,116 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:52,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:52,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:52,415 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:54,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:54,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:56,416 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:24:56,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:24:56,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:24:56,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:24:56,735 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:24:57,546 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:24:57,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:57,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:24:58,417 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:00,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:00,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:00,418 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:02,063 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:02,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:02,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:04,419 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:05,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:05,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:06,420 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:07,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:07,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:07,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:08,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:10,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:25:10,123 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:25:10,123 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:25:10,123 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:25:10,422 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:25:10,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:10,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:11,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:25:11,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:25:11,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:25:12,422 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:12,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:12,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:12,660 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:14,423 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:15,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:15,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:16,424 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:17,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:17,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:18,119 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:20,117 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:25:20,425 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:20,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:20,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:22,426 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:22,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:22,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:23,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:24,427 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:25,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:25,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:26,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:25:26,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:25:26,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:25:27,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:27,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:28,429 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:28,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:30,429 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:30,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:30,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:32,430 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:32,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:32,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:34,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:35,991 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:25:36,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:36,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:36,432 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:38,433 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:38,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:38,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:39,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:40,433 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:40,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:40,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:41,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:25:41,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:25:41,525 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:25:44,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:44,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:44,435 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:44,906 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:46,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:46,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:46,435 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:48,436 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:49,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:49,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:50,119 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:25:50,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:51,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:51,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:52,438 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:54,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:54,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:54,438 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:55,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:25:56,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:56,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:56,439 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:25:56,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:25:56,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:25:56,525 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:25:59,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:25:59,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:00,441 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:00,976 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:01,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:01,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:02,442 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:04,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:04,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:04,443 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:06,327 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:06,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:06,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:08,445 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:09,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:09,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:10,445 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:11,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:11,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:26:11,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:26:11,525 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:26:11,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:11,681 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:12,446 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:15,047 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:26:15,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:15,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:16,447 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:17,043 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:17,069 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:26:17,071 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:26:17,071 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:26:17,071 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:26:17,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:26:17,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:17,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:18,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:20,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:20,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:20,121 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:26:20,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:22,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:23,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:23,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:24,450 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:25,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:25,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:26,451 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:26,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:26:26,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:26:26,525 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:26:27,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:28,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:28,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:28,452 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:30,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:30,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:32,453 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:33,118 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:33,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:33,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:34,454 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:35,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:35,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:36,455 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:38,483 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:38,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:38,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:40,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:40,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:40,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:41,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:26:41,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:26:41,525 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:26:42,457 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:43,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:43,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:43,676 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:44,457 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:45,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:45,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:48,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:48,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:48,459 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:49,195 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:50,122 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:26:50,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:50,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:50,460 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:52,461 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:53,875 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:26:53,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:53,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:54,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:26:56,463 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:56,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:26:56,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:26:56,525 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:26:56,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:56,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:58,463 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:26:58,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:58,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:26:59,894 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:00,464 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:01,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:01,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:03,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:03,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:04,465 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:05,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:06,466 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:07,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:07,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:08,466 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:09,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:09,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:10,639 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:11,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:27:11,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:27:11,525 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:27:12,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:12,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:12,468 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:14,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:14,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:14,469 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:15,988 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:16,469 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:17,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:17,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:19,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:19,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:20,124 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:27:20,471 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:21,125 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:22,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:22,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:22,471 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:24,042 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:27:24,044 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:27:24,044 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:27:24,044 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:27:24,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:24,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:24,472 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:27:24,473 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:26,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:26,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:26,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:27:26,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:27:26,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:27:26,688 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:28,473 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:29,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:29,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:30,474 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:32,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:32,475 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:32,583 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:27:32,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:32,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:34,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:34,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:36,476 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:37,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:37,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:37,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:38,477 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:39,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:39,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:40,477 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:41,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:27:41,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:27:41,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:27:41,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:41,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:42,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:44,479 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:44,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:44,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:46,480 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:46,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:46,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:48,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:48,481 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:49,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:49,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:50,126 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:27:51,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:51,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:52,482 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:53,483 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:27:53,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:53,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:54,483 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:56,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:56,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:56,484 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:27:56,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:27:56,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:27:56,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:27:58,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:58,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:27:58,687 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:00,486 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:00,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:00,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:02,486 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:03,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:03,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:04,212 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:05,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:05,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:06,488 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:08,488 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:08,804 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:28:08,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:08,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:09,582 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:10,490 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:10,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:10,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:11,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:28:11,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:28:11,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:28:13,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:13,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:14,491 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:14,942 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:15,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:15,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:16,492 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:17,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:17,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:18,492 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:20,128 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:28:20,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:20,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:20,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:22,494 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:22,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:22,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:24,495 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:25,648 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:25,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:25,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:26,495 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:26,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:28:26,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:28:26,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:28:27,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:27,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:30,497 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:30,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:30,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:30,988 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:31,014 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:28:31,015 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:28:31,016 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:28:31,016 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:28:31,497 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:28:32,497 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:32,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:33,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:33,498 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:34,498 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:35,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:35,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:36,337 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:37,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:37,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:38,499 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:40,500 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:40,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:40,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:41,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:28:41,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:28:41,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:28:41,653 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:42,501 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:42,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:42,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:44,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:28:45,963 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:28:45,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:45,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:47,115 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:48,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:48,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:50,130 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:28:50,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:50,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:52,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:53,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:53,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:55,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:55,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:56,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:28:56,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:28:56,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:28:57,740 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:28:58,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:28:58,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:00,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:00,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:00,508 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:02,509 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:02,971 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:03,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:03,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:04,510 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:05,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:05,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:07,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:07,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:08,317 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:08,511 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:09,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:10,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:10,512 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:11,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:29:11,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:29:11,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:29:12,513 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:13,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:13,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:13,741 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:15,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:15,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:16,514 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:18,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:18,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:18,515 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:18,994 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:20,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:20,131 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:29:20,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:20,516 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:23,645 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:29:23,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:23,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:24,332 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:24,517 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:26,518 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:26,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:29:26,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:29:26,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:29:26,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:26,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:28,519 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:28,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:28,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:29,670 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:31,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:31,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:32,521 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:33,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:33,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:34,522 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:35,008 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:36,522 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:36,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:36,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:38,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:38,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:40,343 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:40,525 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:41,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:29:41,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:29:41,526 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:29:41,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:41,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:42,526 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:43,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:43,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:45,528 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:45,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:45,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:45,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:48,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:48,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:49,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:29:49,056 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:29:49,057 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:29:49,057 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:29:49,529 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:29:49,529 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:50,133 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:29:50,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:50,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:51,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:51,530 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:53,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:53,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:53,530 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:55,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:55,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:56,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:29:56,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:29:56,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:29:56,527 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:29:57,532 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:29:58,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:58,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:29:59,533 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:01,534 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:01,697 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:02,112 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:30:02,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:02,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:05,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:05,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:05,535 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:07,043 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:07,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:07,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:07,536 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:09,537 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:10,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:10,322 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:11,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:30:11,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:30:11,527 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:30:12,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:12,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:12,743 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:13,539 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:15,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:15,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:15,539 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:17,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:17,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:17,540 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:18,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:20,135 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:30:20,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:20,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:21,542 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:22,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:22,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:23,542 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:23,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:24,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:24,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:25,543 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:26,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:30:26,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:30:26,527 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:30:27,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:27,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:29,057 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:29,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:29,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:29,545 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:31,545 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:32,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:32,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:33,546 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:34,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:34,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:34,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:37,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:37,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:37,548 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:39,549 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:39,742 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:40,810 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:30:40,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:40,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:41,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:30:41,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:30:41,527 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:30:41,549 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:42,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:42,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:45,088 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:45,551 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:45,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:45,804 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:47,551 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:47,823 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:47,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:49,552 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:50,137 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:30:50,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:50,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:50,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:52,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:52,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:53,554 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:55,555 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:55,788 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:30:55,814 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:30:55,815 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:30:55,816 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:30:55,816 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:30:56,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:56,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:56,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:30:56,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:30:56,527 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:30:56,555 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:30:57,555 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:30:58,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:30:58,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:01,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:01,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:01,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:01,556 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:03,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:03,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:03,557 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:05,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:05,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:05,558 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:06,485 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:08,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:08,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:09,559 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:10,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:10,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:11,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:31:11,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:31:11,527 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:31:11,560 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:11,743 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:12,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:12,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:13,561 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:15,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:15,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:17,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:17,562 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:18,658 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:31:18,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:18,688 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:19,563 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:20,139 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:31:20,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:20,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:21,564 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:22,525 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:23,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:23,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:25,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:25,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:25,565 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:26,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:31:26,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:31:26,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:31:27,566 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:27,710 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:28,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:28,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:29,566 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:30,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:30,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:33,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:33,568 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:33,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:33,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:35,568 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:35,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:35,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:37,569 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:38,565 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:38,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:38,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:40,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:40,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:41,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:31:41,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:31:41,527 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:31:41,571 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:43,572 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:43,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:43,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:43,733 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:45,572 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:45,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:45,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:48,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:48,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:49,272 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:49,573 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:50,143 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:31:50,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:50,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:51,574 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:53,575 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:53,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:53,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:54,620 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:31:56,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:31:56,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:31:56,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:31:57,165 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:31:57,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:57,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:31:57,576 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:59,577 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:31:59,972 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:00,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:00,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:01,578 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:02,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:02,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:02,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:32:02,679 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:32:02,680 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:32:02,680 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:32:03,578 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:32:05,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:05,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:05,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:05,579 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:07,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:07,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:07,580 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:09,581 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:10,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:10,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:10,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:11,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:32:11,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:32:11,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:32:12,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:12,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:13,582 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:15,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:15,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:15,583 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:16,042 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:17,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:17,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:17,584 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:20,142 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:32:20,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:20,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:21,143 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:21,585 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:22,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:22,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:23,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:24,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:24,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:25,587 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:26,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:32:26,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:32:26,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:32:26,690 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:27,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:27,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:29,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:29,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:29,588 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:31,589 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:32,090 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:32,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:32,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:33,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:36,077 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:32:36,098 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:36,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:37,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:37,591 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:38,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:38,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:39,592 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:40,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:40,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:41,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:32:41,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:32:41,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:32:41,593 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:42,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:42,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:42,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:45,595 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:45,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:45,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:47,595 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:47,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:47,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:48,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:49,596 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:50,144 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:32:50,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:50,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:52,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:52,787 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:53,494 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:53,597 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:55,598 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:55,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:55,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:56,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:32:56,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:32:56,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:32:57,599 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:32:57,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:57,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:58,691 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:32:59,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:32:59,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:01,600 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:02,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:02,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:03,601 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:04,230 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:04,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:04,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:05,602 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:07,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:07,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:09,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:09,603 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:09,607 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:33:09,609 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:33:09,609 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:33:09,610 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:33:09,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:09,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:10,603 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:33:11,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:33:11,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:33:11,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:33:11,604 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:13,232 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:33:13,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:13,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:13,606 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:14,931 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:16,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:16,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:17,608 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:18,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:18,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:19,608 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:20,146 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:33:20,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:21,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:21,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:21,609 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:23,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:23,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:25,610 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:25,651 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:26,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:26,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:26,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:33:26,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:33:26,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:33:27,611 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:28,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:28,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:29,612 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:31,012 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:31,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:31,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:33,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:33,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:33,613 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:35,614 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:36,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:36,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:36,367 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:37,614 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:38,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:38,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:41,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:41,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:41,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:33:41,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:33:41,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:33:41,616 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:41,708 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:43,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:43,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:43,616 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:45,617 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:46,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:46,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:47,063 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:48,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:48,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:49,618 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:50,147 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:33:51,619 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:51,774 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:33:51,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:51,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:52,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:53,620 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:54,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:54,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:56,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:33:56,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:33:56,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:33:56,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:56,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:57,622 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:57,759 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:33:59,623 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:33:59,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:33:59,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:01,624 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:01,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:01,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:03,134 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:04,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:04,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:05,625 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:06,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:06,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:07,626 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:08,486 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:09,628 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:09,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:09,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:11,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:34:11,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:34:11,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:34:11,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:11,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:13,629 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:13,757 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:14,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:14,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:15,630 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:16,522 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:34:16,523 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:34:16,524 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:34:16,524 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:34:16,630 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:34:16,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:16,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:17,631 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:19,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:19,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:19,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:20,149 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:34:21,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:21,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:21,633 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:23,633 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:24,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:24,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:24,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:25,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:26,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:26,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:26,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:34:26,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:34:26,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:34:29,636 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:29,836 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:34:29,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:29,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:29,884 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:31,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:32,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:32,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:33,638 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:34,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:34,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:35,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:37,639 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:37,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:37,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:39,640 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:39,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:39,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:40,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:41,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:34:41,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:34:41,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:34:41,641 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:42,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:42,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:44,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:44,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:45,642 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:45,950 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:47,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:47,643 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:47,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:49,644 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:49,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:49,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:50,150 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:34:51,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:52,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:52,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:53,645 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:54,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:54,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:55,646 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:56,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:34:56,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:34:56,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:34:56,719 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:34:57,647 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:34:57,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:34:57,985 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:00,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:00,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:01,648 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:02,021 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:02,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:02,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:03,650 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:04,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:04,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:05,650 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:07,369 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:08,405 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:35:08,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:08,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:09,652 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:11,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:11,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:11,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:35:11,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:35:11,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:35:11,653 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:12,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:13,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:13,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:13,654 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:16,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:16,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:17,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:18,057 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:18,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:18,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:19,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:20,152 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:35:21,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:21,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:21,656 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:23,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:23,432 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:35:23,433 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:35:23,434 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:35:23,434 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:35:23,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:23,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:23,657 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:35:25,658 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:26,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:35:26,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:35:26,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:35:26,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:26,556 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:27,658 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:28,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:28,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:28,754 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:29,659 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:31,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:31,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:33,661 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:33,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:33,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:34,121 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:35,662 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:36,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:36,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:37,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:38,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:38,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:39,465 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:41,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:35:41,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:35:41,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:35:41,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:41,664 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:41,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:43,665 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:43,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:43,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:44,807 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:45,666 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:47,143 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:35:47,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:47,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:49,667 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:50,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:50,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:50,154 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:35:50,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:51,668 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:52,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:52,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:55,073 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:55,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:55,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:35:55,669 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:56,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:35:56,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:35:56,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:35:57,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:57,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:35:57,670 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:35:59,671 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:00,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:00,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:00,857 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:02,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:02,121 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:03,672 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:05,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:05,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:05,673 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:06,209 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:07,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:07,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:07,674 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:09,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:10,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:11,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:36:11,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:36:11,529 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:36:11,675 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:11,727 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:12,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:12,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:13,676 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:15,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:15,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:15,677 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:16,913 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:17,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:17,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:19,678 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:20,155 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:36:20,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:20,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:21,679 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:22,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:22,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:22,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:23,680 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:25,722 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:36:25,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:25,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:26,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:36:26,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:36:26,530 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:36:27,604 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:27,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:28,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:28,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:29,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:30,304 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:36:30,306 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:36:30,306 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:36:30,307 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:36:30,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:30,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:30,683 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:36:31,683 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:32,946 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:33,623 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:33,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:35,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:35,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:35,684 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:37,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:38,290 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:38,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:38,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:39,686 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:40,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:40,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:41,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:36:41,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:36:41,530 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:36:43,642 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:43,688 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:43,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:43,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:45,688 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:45,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:45,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:47,689 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:48,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:48,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:48,984 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:50,157 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:36:50,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:50,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:51,691 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:53,692 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:53,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:53,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:54,330 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:55,693 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:36:55,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:55,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:56,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:36:56,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:36:56,530 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:36:58,689 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:58,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:36:59,687 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:36:59,694 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:00,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:00,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:01,695 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:03,696 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:04,165 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:37:04,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:04,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:05,028 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:07,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:07,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:07,697 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:09,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:09,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:09,698 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:10,370 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:11,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:37:11,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:37:11,530 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:37:11,699 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:12,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:12,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:14,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:14,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:15,700 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:15,718 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:17,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:17,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:17,701 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:19,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:19,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:19,702 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:20,159 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:37:21,062 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:22,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:22,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:23,703 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:24,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:24,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:25,704 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:26,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:26,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:37:26,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:37:26,530 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:37:27,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:27,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:27,704 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:29,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:29,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:31,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:31,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:31,706 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:31,754 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:33,707 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:34,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:34,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:35,708 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:36,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:36,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:37,104 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:37,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:37:37,132 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:37:37,132 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:37:37,133 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:37:37,709 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:37:39,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:39,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:39,709 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:41,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:37:41,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:37:41,530 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:37:41,710 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:42,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:42,732 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:37:42,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:42,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:43,711 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:44,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:44,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:47,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:47,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:47,712 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:47,802 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:49,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:49,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:49,712 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:50,161 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:37:51,714 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:52,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:52,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:53,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:54,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:54,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:55,715 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:56,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:37:56,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:37:56,530 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:37:56,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:56,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:57,716 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:37:58,503 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:37:59,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:59,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:37:59,716 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:01,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:01,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:03,718 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:03,832 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:04,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:04,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:05,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:06,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:06,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:07,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:09,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:09,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:09,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:11,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:38:11,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:38:11,531 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:38:11,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:11,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:11,721 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:13,722 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:14,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:14,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:14,513 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:15,723 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:16,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:16,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:19,724 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:19,854 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:19,931 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:38:19,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:19,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:20,162 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:38:21,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:22,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:22,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:23,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:24,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:24,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:25,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:26,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:38:26,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:38:26,531 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:38:27,727 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:27,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:27,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:29,727 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:29,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:29,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:30,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:31,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:32,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:32,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:34,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:34,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:35,729 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:35,891 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:37,729 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:38,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:38,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:39,730 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:40,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:40,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:41,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:41,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:38:41,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:38:41,531 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:38:42,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:42,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:43,731 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:43,938 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:38:43,940 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:38:43,940 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:38:43,940 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:38:44,732 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:38:45,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:45,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:45,732 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:46,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:47,733 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:48,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:48,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:49,733 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:38:50,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:50,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:50,163 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:38:52,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:52,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:52,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:54,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:54,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:56,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:38:56,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:38:56,531 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:38:57,735 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:38:57,904 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:38:57,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:38:57,933 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:00,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:00,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:02,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:02,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:02,736 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:03,738 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:05,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:05,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:07,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:07,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:07,739 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:08,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:09,739 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:10,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:10,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:11,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:39:11,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:39:11,531 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:39:11,741 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:12,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:12,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:13,725 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:14,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:14,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:15,742 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:17,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:17,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:17,742 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:19,057 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:19,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:19,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:19,743 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:20,165 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:39:22,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:22,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:23,744 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:24,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:24,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:24,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:25,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:26,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:39:26,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:39:26,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:39:27,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:27,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:27,746 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:29,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:29,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:29,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:31,747 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:32,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:32,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:33,747 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:35,051 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:35,748 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:36,220 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:39:36,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:36,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:39,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:39,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:39,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:40,382 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:41,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:41,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:41,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:39:41,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:39:41,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:39:41,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:43,750 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:44,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:44,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:45,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:46,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:46,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:47,751 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:49,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:49,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:49,752 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:50,167 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:39:51,034 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:51,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:51,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:51,753 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:54,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:54,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:55,754 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:56,377 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:39:56,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:39:56,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:39:56,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:39:56,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:56,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:57,754 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:39:59,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:59,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:39:59,755 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:01,713 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:01,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:01,740 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:40:01,741 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:40:01,742 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:40:01,742 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:40:01,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:01,756 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:40:03,757 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:03,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:03,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:05,757 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:06,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:06,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:07,052 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:07,758 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:08,590 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:08,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:11,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:40:11,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:40:11,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:40:11,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:11,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:11,759 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:12,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:13,760 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:15,374 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:40:15,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:15,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:15,760 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:17,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:17,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:17,745 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:19,762 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:20,169 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:40:20,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:20,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:21,762 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:22,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:22,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:23,081 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:23,763 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:25,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:25,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:26,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:40:26,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:40:26,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:40:27,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:27,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:27,764 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:28,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:29,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:30,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:30,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:31,766 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:32,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:32,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:33,777 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:35,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:35,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:35,766 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:37,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:37,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:37,768 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:39,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:39,768 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:40,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:40,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:41,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:40:41,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:40:41,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:40:42,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:42,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:43,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:44,468 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:45,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:45,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:45,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:47,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:47,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:47,771 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:49,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:49,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:49,806 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:50,170 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:40:51,772 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:52,959 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:40:52,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:52,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:53,773 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:55,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:40:55,773 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:40:55,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:55,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:56,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:40:56,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:40:56,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:40:57,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:57,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:40:59,774 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:00,486 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:00,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:00,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:01,775 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:02,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:02,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:03,775 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:04,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:04,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:05,825 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:07,777 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:08,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:08,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:08,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:41:08,528 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:41:08,528 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:41:08,529 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:41:08,777 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:41:09,778 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:10,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:10,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:11,172 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:11,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:41:11,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:41:11,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:41:11,778 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:12,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:12,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:15,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:15,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:15,780 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:16,507 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:17,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:17,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:17,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:19,782 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:20,173 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:41:20,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:20,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:21,842 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:22,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:22,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:23,783 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:25,783 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:25,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:25,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:26,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:41:26,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:41:26,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:41:27,179 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:27,784 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:27,846 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:27,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:31,382 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:41:31,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:31,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:31,786 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:32,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:33,787 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:34,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:34,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:35,787 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:36,430 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:36,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:37,865 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:39,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:39,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:39,789 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:41,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:41,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:41,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:41:41,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:41:41,533 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:41:41,789 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:43,192 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:43,790 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:44,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:44,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:46,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:46,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:47,791 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:48,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:49,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:49,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:49,791 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:50,174 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:41:51,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:51,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:51,792 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:53,873 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:54,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:54,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:55,793 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:56,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:41:56,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:41:56,532 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:41:56,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:56,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:57,794 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:59,209 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:41:59,795 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:41:59,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:41:59,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:01,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:01,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:03,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:03,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:03,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:04,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:05,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:06,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:06,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:07,802 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:09,885 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:10,525 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:42:10,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:10,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:11,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:42:11,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:42:11,533 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:42:11,803 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:12,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:12,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:13,804 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:15,210 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:15,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:42:15,238 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:42:15,238 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:42:15,239 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:42:15,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:15,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:15,804 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:42:15,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:17,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:17,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:19,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:19,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:19,806 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:20,176 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:42:20,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:21,807 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:22,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:22,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:23,807 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:24,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:24,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:25,888 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:26,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:42:26,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:42:26,533 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:42:27,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:27,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:27,809 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:29,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:29,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:29,810 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:31,224 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:31,810 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:32,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:32,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:34,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:34,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:35,812 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:36,557 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:37,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:37,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:37,812 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:39,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:39,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:39,813 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:41,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:42:41,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:42:41,534 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:42:41,757 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:42,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:42,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:43,814 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:44,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:44,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:45,815 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:47,218 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:47,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:47,837 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:42:47,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:47,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:50,177 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:42:50,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:50,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:51,817 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:52,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:52,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:52,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:53,817 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:55,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:55,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:55,818 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:42:56,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:42:56,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:42:56,533 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:42:57,749 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:42:57,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:57,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:42:59,819 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:00,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:00,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:01,821 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:02,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:02,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:03,219 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:03,821 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:05,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:05,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:07,822 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:07,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:07,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:08,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:09,823 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:10,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:10,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:11,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:43:11,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:43:11,534 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:43:11,823 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:12,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:12,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:13,730 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:15,825 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:15,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:15,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:17,825 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:17,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:17,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:19,214 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:19,826 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:20,179 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:43:20,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:20,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:21,914 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:43:21,916 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:43:21,917 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:43:21,917 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:43:22,827 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:43:22,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:22,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:23,827 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:24,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:25,828 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:26,461 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:43:26,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:26,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:26,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:43:26,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:43:26,534 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:43:27,829 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:29,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:29,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:29,895 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:31,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:31,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:31,830 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:33,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:34,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:34,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:35,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:35,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:36,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:36,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:39,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:39,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:39,833 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:40,559 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:41,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:41,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:43:41,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:43:41,534 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:43:41,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:41,834 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:43,834 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:44,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:44,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:45,896 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:46,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:46,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:47,836 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:49,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:49,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:49,836 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:50,180 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:43:51,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:51,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:51,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:51,837 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:54,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:54,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:55,838 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:56,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:43:56,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:43:56,535 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:43:56,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:43:56,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:56,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:57,839 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:43:59,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:59,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:43:59,840 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:01,678 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:01,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:01,901 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:03,841 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:05,028 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:44:05,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:05,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:05,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:07,226 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:07,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:07,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:07,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:10,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:10,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:11,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:44:11,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:44:11,535 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:44:11,844 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:12,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:12,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:12,566 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:13,844 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:14,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:14,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:15,845 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:16,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:16,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:17,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:19,847 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:19,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:19,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:20,182 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:44:21,848 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:21,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:21,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:23,221 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:23,848 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:24,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:24,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:26,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:44:26,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:44:26,535 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:44:26,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:26,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:27,849 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:28,562 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:28,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:44:28,590 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:44:28,590 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:44:28,591 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:44:28,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:44:28,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:29,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:29,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:31,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:31,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:31,852 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:33,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:33,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:33,902 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:36,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:36,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:36,853 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:38,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:38,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:38,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:39,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:40,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:41,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:44:41,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:44:41,536 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:44:42,280 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:44:42,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:42,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:44,557 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:44,856 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:45,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:45,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:46,857 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:47,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:47,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:48,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:49,900 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:50,183 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:44:50,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:50,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:52,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:52,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:52,859 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:54,859 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:55,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:55,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:55,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:44:56,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:44:56,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:44:56,535 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:44:56,860 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:44:57,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:44:57,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:00,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:00,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:00,568 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:00,861 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:02,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:02,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:02,862 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:04,863 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:05,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:05,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:05,908 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:07,206 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:07,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:08,864 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:10,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:10,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:10,864 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:11,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:11,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:45:11,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:45:11,535 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:45:12,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:12,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:12,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:15,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:15,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:16,595 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:16,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:17,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:17,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:18,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:20,185 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:45:20,868 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:20,874 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:45:20,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:20,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:21,914 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:23,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:23,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:24,870 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:25,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:25,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:26,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:45:26,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:45:26,535 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:45:26,871 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:27,239 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:28,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:28,793 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:28,873 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:30,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:30,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:32,579 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:32,873 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:33,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:33,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:34,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:35,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:45:35,275 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:45:35,275 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:45:35,275 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:45:35,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:35,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:35,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:45:35,875 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:36,875 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:37,917 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:38,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:38,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:40,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:40,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:40,877 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:41,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:45:41,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:45:41,535 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:45:42,877 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:43,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:43,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:43,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:44,878 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:45,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:45,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:48,596 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:48,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:48,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:48,879 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:50,187 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:45:50,880 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:50,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:50,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:52,881 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:53,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:53,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:54,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:45:55,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:55,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:56,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:45:56,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:45:56,535 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:45:56,882 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:58,882 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:45:59,077 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:45:59,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:59,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:45:59,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:00,883 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:01,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:01,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:03,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:03,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:04,884 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:04,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:06,885 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:06,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:06,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:08,886 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:09,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:09,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:10,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:11,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:11,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:11,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:46:11,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:46:11,536 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:46:12,887 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:13,827 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:13,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:14,887 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:15,587 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:15,859 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:15,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:16,888 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:17,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:17,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:20,188 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:46:20,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:20,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:20,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:20,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:22,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:22,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:22,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:24,891 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:25,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:25,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:26,297 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:26,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:46:26,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:46:26,536 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:46:27,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:27,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:28,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:30,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:30,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:30,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:31,637 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:32,894 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:32,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:32,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:36,470 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:46:36,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:36,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:36,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:36,978 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:38,896 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:39,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:39,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:40,897 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:41,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:41,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:41,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:46:41,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:46:41,536 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:46:42,327 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:42,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:46:42,354 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:46:42,354 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:46:42,354 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:46:42,898 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:46:43,898 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:44,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:44,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:44,898 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:46,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:46,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:46,899 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:47,681 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:48,900 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:49,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:49,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:50,191 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:46:51,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:51,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:52,901 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:53,030 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:54,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:54,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:54,901 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:56,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:46:56,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:46:56,536 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:46:56,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:56,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:56,902 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:46:58,375 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:46:59,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:46:59,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:00,903 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:01,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:01,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:02,904 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:03,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:04,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:04,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:04,905 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:06,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:06,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:08,905 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:09,060 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:09,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:09,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:10,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:11,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:11,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:11,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:47:11,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:47:11,536 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:47:14,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:14,907 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:14,987 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:47:15,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:15,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:16,908 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:17,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:17,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:18,908 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:19,760 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:20,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:20,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:20,192 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:47:20,909 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:22,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:22,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:24,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:24,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:24,910 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:25,105 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:26,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:47:26,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:47:26,536 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:47:26,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:26,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:26,911 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:28,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:28,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:28,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:30,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:31,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:31,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:32,913 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:33,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:33,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:34,914 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:35,798 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:36,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:36,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:38,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:38,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:38,915 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:40,916 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:41,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:41,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:47:41,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:47:41,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:47:41,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:41,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:42,916 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:43,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:43,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:46,504 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:46,917 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:46,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:46,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:48,918 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:48,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:48,967 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:49,205 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:47:49,206 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:47:49,207 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:47:49,207 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:47:49,918 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:47:49,919 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:50,193 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:47:50,919 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:51,856 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:52,426 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:47:52,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:52,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:54,920 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:55,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:55,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:56,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:47:56,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:47:56,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:47:56,921 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:47:57,204 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:47:57,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:57,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:47:58,922 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:00,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:00,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:02,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:02,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:02,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:02,923 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:04,924 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:05,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:05,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:06,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:07,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:07,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:07,885 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:10,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:10,689 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:10,926 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:11,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:48:11,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:48:11,537 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:48:12,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:12,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:12,926 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:13,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:14,928 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:15,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:15,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:17,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:17,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:18,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:18,929 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:20,195 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:48:20,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:20,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:20,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:22,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:22,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:22,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:23,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:24,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:24,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:26,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:48:26,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:48:26,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:48:26,931 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:27,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:27,419 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:28,932 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:29,282 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:30,933 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:31,059 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:48:31,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:31,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:33,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:33,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:34,635 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:34,934 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:35,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:35,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:36,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:37,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:37,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:38,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:39,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:39,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:39,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:41,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:48:41,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:48:41,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:48:42,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:42,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:42,936 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:44,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:44,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:44,937 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:45,325 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:46,938 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:47,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:47,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:49,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:49,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:50,197 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:48:50,677 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:50,939 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:52,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:52,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:52,940 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:54,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:54,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:54,941 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:48:56,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:48:56,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:48:56,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:48:56,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:48:57,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:57,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:59,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:48:59,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:01,770 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:02,014 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:02,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:04,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:04,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:06,770 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:07,018 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:49:07,020 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:49:07,020 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:49:07,020 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:49:07,483 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:49:07,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:07,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:07,945 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:49:08,946 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:10,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:10,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:10,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:11,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:49:11,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:49:11,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:49:12,308 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:12,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:12,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:14,948 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:15,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:15,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:16,948 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:17,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:17,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:17,637 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:18,950 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:20,198 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:49:20,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:20,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:22,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:22,585 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:22,951 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:22,967 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:24,952 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:25,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:25,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:26,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:49:26,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:49:26,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:49:26,952 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:27,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:27,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:28,304 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:30,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:30,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:30,954 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:32,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:32,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:32,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:33,644 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:34,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:35,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:35,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:37,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:37,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:38,956 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:38,980 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:40,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:40,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:40,958 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:41,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:49:41,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:49:41,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:49:42,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:42,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:42,958 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:44,310 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:46,443 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:49:46,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:46,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:46,960 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:48,960 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:49,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:49,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:49,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:50,200 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:49:50,961 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:51,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:51,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:54,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:54,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:54,963 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:54,974 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:49:56,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:56,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:56,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:49:56,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:49:56,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:49:56,963 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:49:58,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:58,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:49:58,964 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:00,314 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:01,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:01,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:02,965 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:03,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:03,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:04,966 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:05,645 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:06,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:06,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:06,967 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:08,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:08,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:10,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:10,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:10,968 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:10,969 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:11,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:50:11,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:50:11,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:50:12,969 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:13,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:13,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:13,662 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:50:13,664 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:50:13,665 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:50:13,665 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:50:13,969 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:50:14,970 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:15,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:15,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:16,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:18,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:18,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:18,972 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:20,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:20,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:20,201 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:50:20,972 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:21,640 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:22,973 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:23,571 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:50:23,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:23,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:26,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:26,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:26,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:50:26,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:50:26,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:50:26,683 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:26,975 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:28,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:28,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:28,975 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:30,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:31,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:31,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:32,308 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:33,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:33,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:34,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:36,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:36,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:36,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:37,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:38,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:38,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:38,979 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:41,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:41,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:41,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:50:41,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:50:41,539 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:50:42,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:42,980 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:43,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:43,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:44,981 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:46,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:46,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:46,981 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:48,287 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:48,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:48,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:50,202 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:50:50,982 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:51,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:51,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:52,983 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:53,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:53,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:53,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:54,984 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:50:56,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:56,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:56,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:50:56,538 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:50:56,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:50:58,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:58,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:50:58,676 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:50:58,986 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:00,986 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:01,896 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:51:01,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:01,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:02,987 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:04,276 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:04,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:04,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:06,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:06,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:06,989 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:08,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:09,590 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:09,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:09,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:10,990 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:11,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:51:11,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:51:11,539 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:51:11,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:11,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:14,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:14,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:14,919 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:14,992 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:16,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:16,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:16,993 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:18,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:19,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:19,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:20,204 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:51:20,206 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:20,271 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:51:20,272 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:51:20,272 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:51:20,273 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:51:20,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:51:21,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:21,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:22,996 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:24,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:24,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:24,997 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:25,575 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:26,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:51:26,539 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:51:26,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:51:26,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:26,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:26,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:29,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:29,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:30,919 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:31,000 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:32,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:32,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:33,001 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:34,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:34,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:35,001 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:36,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:36,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:36,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:39,003 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:40,451 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:51:40,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:40,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:41,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:41,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:41,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:51:41,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:51:41,539 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:51:43,006 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:43,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:43,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:45,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:45,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:46,899 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:47,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:48,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:48,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:49,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:50,205 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:51:50,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:50,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:51,009 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:52,207 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:53,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:53,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:55,010 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:55,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:55,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:56,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:51:56,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:51:56,540 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:51:57,011 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:51:57,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:51:58,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:58,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:51:59,012 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:00,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:00,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:02,849 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:03,014 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:03,677 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:03,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:05,015 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:05,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:05,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:07,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:08,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:08,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:08,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:10,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:10,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:11,017 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:11,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:52:11,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:52:11,539 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:52:13,018 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:13,509 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:13,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:13,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:15,019 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:15,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:15,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:17,019 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:18,838 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:19,458 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:52:19,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:19,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:20,207 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:52:21,021 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:22,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:22,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:23,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:24,157 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:24,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:24,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:25,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:26,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:52:26,539 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:52:26,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:52:26,843 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:52:26,845 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:52:26,845 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:52:26,846 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:52:27,023 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:52:27,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:27,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:29,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:29,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:29,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:29,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:31,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:32,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:32,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:33,025 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:34,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:34,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:34,810 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:37,026 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:37,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:37,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:39,028 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:39,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:39,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:40,135 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:41,028 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:41,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:52:41,540 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:52:41,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:52:42,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:42,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:44,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:44,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:45,030 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:45,454 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:47,030 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:47,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:47,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:49,031 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:49,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:49,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:50,208 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:52:50,786 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:52,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:52,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:53,032 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:54,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:54,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:55,033 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:56,101 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:52:56,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:52:56,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:52:56,540 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:52:57,034 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:52:58,017 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:52:58,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:52:58,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:00,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:00,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:01,035 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:01,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:02,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:03,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:03,036 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:05,037 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:05,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:05,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:06,753 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:07,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:07,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:09,037 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:10,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:10,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:11,038 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:11,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:53:11,540 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:53:11,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:53:12,087 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:12,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:12,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:13,039 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:15,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:15,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:17,040 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:17,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:17,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:17,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:19,041 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:20,210 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:53:20,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:20,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:21,041 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:22,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:23,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:23,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:25,043 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:25,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:25,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:26,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:53:26,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:53:26,540 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:53:27,043 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:27,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:27,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:28,072 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:29,044 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:30,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:30,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:33,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:33,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:33,045 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:33,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:33,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:53:33,416 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:53:33,416 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:53:33,417 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:53:34,045 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:53:35,046 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:36,521 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:53:36,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:36,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:37,047 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:38,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:39,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:39,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:41,048 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:41,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:41,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:41,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:53:41,540 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:53:41,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:53:43,048 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:44,067 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:44,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:44,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:45,050 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:46,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:46,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:49,051 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:49,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:49,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:49,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:50,212 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:53:51,052 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:51,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:51,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:53,052 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:53,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:53,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:54,723 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:53:56,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:56,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:56,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:53:56,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:53:56,540 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:53:57,054 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:53:58,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:58,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:53:59,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:00,059 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:01,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:01,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:01,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:03,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:03,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:05,056 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:05,379 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:06,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:06,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:07,058 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:08,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:08,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:09,058 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:10,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:11,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:11,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:11,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:54:11,540 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:54:11,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:54:13,060 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:15,037 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:54:15,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:15,060 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:15,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:16,039 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:17,061 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:18,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:18,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:20,214 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:54:20,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:20,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:21,063 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:21,216 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:23,063 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:23,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:23,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:25,064 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:25,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:25,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:26,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:26,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:54:26,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:54:26,541 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:54:28,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:28,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:29,066 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:30,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:30,597 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:31,067 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:32,013 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:33,067 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:33,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:33,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:35,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:35,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:37,068 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:37,345 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:38,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:38,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:39,069 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:40,034 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:54:40,035 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:54:40,036 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:54:40,036 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:54:40,069 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:54:40,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:40,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:41,070 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:41,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:54:41,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:54:41,541 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:54:42,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:43,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:43,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:45,072 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:45,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:45,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:47,072 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:47,998 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:48,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:48,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:49,073 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:50,215 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:54:50,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:50,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:53,075 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:53,340 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:54,472 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:54:54,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:54,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:55,076 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:56,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:54:56,541 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:54:56,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:54:57,077 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:54:57,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:57,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:58,666 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:54:59,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:54:59,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:01,078 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:02,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:02,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:03,079 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:03,997 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:04,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:04,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:05,080 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:06,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:06,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:09,081 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:09,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:09,321 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:09,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:11,082 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:11,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:11,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:11,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:55:11,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:55:11,541 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:55:13,082 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:13,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:13,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:14,655 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:16,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:16,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:17,084 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:18,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:18,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:19,085 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:19,996 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:20,216 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:55:20,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:20,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:21,085 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:23,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:23,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:25,087 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:25,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:25,324 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:25,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:26,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:55:26,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:55:26,542 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:55:27,088 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:28,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:28,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:29,089 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:30,661 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:32,024 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:55:32,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:32,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:33,090 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:34,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:34,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:35,091 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:35,989 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:37,092 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:37,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:37,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:39,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:39,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:41,093 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:41,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:41,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:41,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:41,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:55:41,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:55:41,542 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:55:43,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:44,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:44,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:45,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:46,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:46,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:46,658 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:46,683 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:55:46,685 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:55:46,685 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:55:46,686 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:55:47,095 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:55:49,096 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:49,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:49,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:50,218 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:55:51,097 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:51,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:51,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:51,981 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:53,098 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:54,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:54,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:56,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:56,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:56,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:55:56,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:55:56,542 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:55:57,099 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:57,309 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:55:59,100 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:55:59,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:55:59,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:01,101 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:01,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:01,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:02,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:03,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:03,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:05,102 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:06,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:06,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:07,103 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:07,963 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:09,104 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:09,433 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:56:10,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:10,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:11,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:56:11,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:56:11,542 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:56:12,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:12,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:13,106 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:13,298 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:15,107 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:15,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:15,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:17,108 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:17,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:17,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:18,623 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:20,220 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:56:20,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:20,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:21,109 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:22,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:22,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:23,110 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:23,943 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:24,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:24,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:25,110 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:26,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:56:26,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:56:26,542 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:56:27,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:27,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:29,112 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:29,273 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:29,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:29,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:31,112 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:31,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:31,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:33,113 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:34,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:34,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:34,591 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:36,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:36,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:37,115 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:38,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:38,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:39,116 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:39,917 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:40,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:40,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:41,116 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:41,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:56:41,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:56:41,542 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:56:42,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:42,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:45,118 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:45,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:46,452 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:56:46,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:46,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:47,119 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:49,120 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:49,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:49,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:50,222 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:56:50,571 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:51,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:51,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:53,121 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:53,262 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:56:53,263 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:56:53,263 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:56:53,264 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:56:53,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:53,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:54,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:56:55,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:55,896 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:56:56,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:56,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:56,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:56:56,542 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:56:56,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:56:57,123 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:56:58,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:58,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:56:59,123 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:01,219 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:01,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:01,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:03,125 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:03,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:03,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:05,126 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:06,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:06,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:06,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:07,126 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:08,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:08,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:11,128 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:11,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:11,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:11,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:57:11,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:57:11,543 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:57:11,755 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:13,129 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:13,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:13,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:15,129 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:16,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:16,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:17,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:18,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:18,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:19,130 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:20,223 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:57:21,131 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:21,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:21,536 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:22,224 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:23,132 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:24,058 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:57:24,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:24,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:26,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:26,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:26,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:57:26,542 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:57:26,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:57:27,133 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:27,721 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:28,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:28,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:29,134 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:31,135 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:31,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:31,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:33,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:33,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:33,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:35,137 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:36,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:36,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:37,137 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:38,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:38,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:38,447 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:39,138 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:41,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:41,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:41,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:57:41,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:57:41,543 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:57:43,140 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:43,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:43,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:43,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:45,141 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:45,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:46,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:47,141 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:48,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:48,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:49,090 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:50,225 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:57:51,143 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:51,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:51,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:53,144 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:53,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:53,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:54,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:55,145 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:55,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:55,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:56,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:57:56,543 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:57:56,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:57:58,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:58,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:57:59,146 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:57:59,729 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:57:59,755 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:57:59,757 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:57:59,757 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:57:59,757 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:58:00,147 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:58:01,147 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:02,037 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:58:02,057 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:02,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:03,148 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:04,090 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:04,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:05,046 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:06,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:06,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:07,149 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:08,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:08,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:09,149 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:10,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:11,151 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:11,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:58:11,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:58:11,543 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:58:11,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:11,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:13,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:13,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:15,152 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:15,686 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:16,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:16,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:17,152 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:18,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:18,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:19,153 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:20,227 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:58:21,004 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:21,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:21,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:23,154 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:23,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:23,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:25,155 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:26,332 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:26,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:58:26,543 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:58:26,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:58:27,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:27,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:27,155 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:29,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:29,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:31,157 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:31,651 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:31,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:32,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:33,157 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:34,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:34,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:35,158 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:36,972 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:37,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:37,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:39,159 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:40,616 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:58:40,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:40,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:41,160 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:41,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:58:41,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:58:41,543 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:58:42,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:42,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:42,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:43,160 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:45,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:45,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:47,162 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:47,523 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:47,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:47,617 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:49,162 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:50,229 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:58:50,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:50,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:51,163 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:52,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:52,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:52,932 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:55,164 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:55,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:55,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:56,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:58:56,543 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:58:56,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:58:57,165 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:58:57,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:57,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:58,763 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:58:59,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:58:59,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:01,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:01,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:03,763 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:04,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:04,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:06,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:06,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:08,764 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:09,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:09,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:11,170 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:11,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:11,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:11,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:59:11,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:59:11,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:59:13,170 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:14,076 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:14,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:14,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:15,171 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:17,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 00:59:17,415 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 00:59:17,416 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 00:59:17,416 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 00:59:17,887 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:59:17,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:17,923 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:18,173 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 00:59:19,173 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:19,418 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:20,230 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:59:20,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:20,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:22,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:22,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:25,224 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:25,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:25,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:26,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:59:26,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:59:26,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:59:27,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:27,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:29,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:29,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:30,926 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:31,177 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:31,893 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:31,903 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:34,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:34,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:35,178 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:36,507 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:36,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:36,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:39,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:39,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:41,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:41,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:41,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:41,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:59:41,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:59:41,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:59:42,180 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:44,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:44,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:46,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:46,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:46,748 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:48,181 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:49,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:49,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:50,232 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 00:59:51,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:51,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:52,327 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:54,183 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 00:59:54,784 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 00:59:54,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:54,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:56,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 00:59:56,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 00:59:56,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 00:59:57,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:57,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:58,000 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 00:59:59,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 00:59:59,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:00,185 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:02,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:02,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:03,575 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:04,186 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:04,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:04,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:07,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:07,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:09,221 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:09,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:09,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:10,188 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:11,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:00:11,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:00:11,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:00:12,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:12,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:14,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:14,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:14,808 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:16,190 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:16,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:17,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:19,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:19,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:20,233 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:00:20,235 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:21,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:21,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:22,192 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:23,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:23,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:26,057 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:26,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:00:26,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:00:26,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:00:26,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:26,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:28,194 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:28,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:28,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:31,693 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:32,185 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:00:32,195 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:32,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:32,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:34,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:34,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:36,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:36,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:37,293 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:38,197 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:39,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:39,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:41,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:00:41,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:00:41,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:00:41,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:41,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:42,857 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:44,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:44,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:44,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:46,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:46,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:48,456 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:49,797 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:49,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:50,200 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:50,235 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:00:51,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:51,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:54,129 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:00:54,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:54,598 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:56,202 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:00:56,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:00:56,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:00:56,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:00:56,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:56,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:59,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:59,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:00:59,714 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:00,204 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:01,534 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:01,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:04,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:04,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:05,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:06,206 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:06,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:06,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:09,649 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:01:09,667 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:09,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:10,944 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:11,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:01:11,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:01:11,544 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:01:12,207 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:12,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:12,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:14,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:14,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:16,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:17,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:17,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:18,209 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:19,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:19,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:20,236 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:01:21,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:21,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:22,176 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:23,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:24,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:24,211 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:26,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:01:26,545 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:01:26,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:01:26,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:26,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:27,786 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:28,212 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:28,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:28,961 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:31,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:31,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:33,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:33,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:33,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:34,214 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:36,726 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:36,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:38,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:38,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:39,035 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:40,215 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:41,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:01:41,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:01:41,545 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:01:41,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:41,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:43,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:43,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:44,641 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:46,217 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:46,788 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:01:46,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:46,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:49,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:49,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:50,238 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:01:50,240 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:51,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:51,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:52,219 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:54,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:54,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:55,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:01:56,220 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:01:56,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:56,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:56,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:01:56,545 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:01:56,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:01:59,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:01:59,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:01,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:01,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:01,495 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:02,221 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:04,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:04,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:06,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:06,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:07,117 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:08,223 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:08,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:08,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:11,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:11,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:11,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:02:11,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:02:11,546 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:02:12,785 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:13,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:13,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:14,225 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:15,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:16,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:18,371 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:18,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:18,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:20,227 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:20,239 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:02:20,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:20,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:23,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:23,996 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:02:24,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:24,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:24,228 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:26,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:02:26,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:02:26,545 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:02:26,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:26,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:28,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:28,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:29,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:30,230 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:31,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:31,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:33,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:33,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:35,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:36,231 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:36,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:36,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:38,712 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:38,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:40,769 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:41,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:02:41,545 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:02:41,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:02:41,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:41,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:42,233 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:43,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:43,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:45,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:45,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:46,385 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:48,235 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:48,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:48,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:50,241 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:02:50,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:50,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:51,940 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:52,236 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:02:53,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:53,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:55,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:55,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:56,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:02:56,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:02:56,546 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:02:57,695 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:02:57,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:57,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:02:58,238 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:01,316 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:03:01,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:01,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:03,137 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:04,239 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:04,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:04,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:06,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:06,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:08,764 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:09,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:09,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:10,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:11,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:11,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:11,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:03:11,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:03:11,546 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:03:13,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:13,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:14,325 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:15,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:15,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:16,243 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:18,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:18,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:19,919 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:20,243 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:03:20,246 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:20,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:20,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:23,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:23,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:25,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:25,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:25,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:26,247 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:26,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:03:26,546 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:03:26,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:03:28,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:28,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:30,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:30,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:31,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:32,249 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:33,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:33,231 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:35,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:35,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:36,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:38,251 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:38,485 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:03:38,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:38,513 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:41,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:41,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:41,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:03:41,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:03:41,546 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:03:41,742 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:43,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:43,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:44,252 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:46,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:46,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:46,742 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:48,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:48,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:48,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:50,245 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:03:51,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:51,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:52,246 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:53,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:53,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:54,255 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:03:56,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:56,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:56,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:03:56,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:03:56,546 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:03:57,714 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:03:58,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:03:58,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:00,256 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:01,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:01,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:03,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:03,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:03,285 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:06,258 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:06,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:06,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:08,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:08,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:08,950 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:11,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:11,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:11,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:04:11,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:04:11,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:04:12,260 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:13,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:13,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:14,692 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:16,261 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:16,646 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:04:16,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:16,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:19,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:19,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:20,109 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:20,249 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:04:21,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:21,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:22,265 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:24,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:24,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:25,699 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:26,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:26,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:26,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:04:26,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:04:26,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:04:28,267 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:29,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:29,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:30,734 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:31,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:31,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:34,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:34,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:34,268 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:36,012 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:36,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:36,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:38,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:38,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:40,270 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:40,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:40,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:41,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:41,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:04:41,546 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:04:41,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:04:43,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:43,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:45,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:45,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:46,271 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:47,206 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:48,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:48,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:50,250 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:04:50,273 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:50,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:50,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:52,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:53,953 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:04:53,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:53,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:56,275 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:04:56,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:04:56,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:04:56,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:04:56,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:56,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:57,687 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:04:58,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:04:58,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:01,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:01,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:02,277 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:03,058 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:03,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:03,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:06,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:06,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:08,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:08,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:08,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:08,608 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:11,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:11,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:11,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:05:11,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:05:11,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:05:12,279 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:13,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:13,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:13,698 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:16,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:16,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:18,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:18,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:18,281 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:18,782 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:20,252 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:05:21,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:21,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:23,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:23,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:24,282 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:24,423 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:26,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:26,319 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:26,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:05:26,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:05:26,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:05:28,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:28,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:30,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:30,284 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:31,541 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:05:31,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:31,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:34,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:34,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:35,640 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:36,286 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:36,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:36,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:39,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:39,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:40,288 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:41,190 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:41,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:41,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:41,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:05:41,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:05:41,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:05:44,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:44,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:46,289 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:46,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:46,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:46,813 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:49,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:49,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:50,254 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:05:51,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:51,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:52,291 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:52,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:54,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:54,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:56,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:56,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:56,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:05:56,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:05:56,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:05:57,699 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:05:58,292 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:05:59,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:05:59,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:01,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:01,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:03,650 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:04,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:04,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:04,294 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:06,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:06,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:08,296 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:09,250 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:09,394 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:06:09,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:09,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:11,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:06:11,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:06:11,548 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:06:12,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:12,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:14,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:14,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:14,297 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:14,830 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:17,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:17,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:19,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:19,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:20,256 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:06:20,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:20,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:22,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:22,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:24,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:24,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:26,030 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:26,300 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:26,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:06:26,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:06:26,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:06:26,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:26,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:29,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:29,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:31,697 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:32,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:32,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:32,302 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:34,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:34,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:36,303 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:36,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:36,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:37,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:38,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:38,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:41,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:06:41,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:06:41,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:06:41,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:41,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:42,305 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:42,686 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:43,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:43,744 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:47,095 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:06:47,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:47,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:48,307 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:48,377 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:49,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:49,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:50,258 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:06:51,863 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:51,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:53,961 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:54,309 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:06:54,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:54,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:56,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:06:56,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:06:56,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:06:56,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:56,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:59,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:06:59,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:06:59,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:00,311 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:01,763 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:01,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:04,312 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:04,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:04,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:05,131 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:06,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:06,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:09,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:09,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:10,314 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:10,772 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:11,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:11,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:11,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:07:11,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:07:11,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:07:14,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:14,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:16,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:16,242 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:16,315 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:16,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:19,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:19,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:20,260 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:07:21,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:21,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:22,261 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:22,317 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:24,609 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:07:24,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:24,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:26,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:07:26,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:07:26,548 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:07:27,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:27,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:27,724 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:28,319 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:29,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:29,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:32,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:32,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:32,320 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:33,087 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:34,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:34,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:37,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:37,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:38,322 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:38,690 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:39,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:39,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:41,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:07:41,547 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:07:41,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:07:42,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:42,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:44,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:44,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:44,265 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:44,323 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:47,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:47,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:49,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:49,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:49,893 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:50,261 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:07:50,325 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:52,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:52,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:54,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:54,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:55,479 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:07:56,326 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:07:56,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:07:56,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:07:56,548 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:07:56,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:56,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:58,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:07:58,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:00,328 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:01,039 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:02,150 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:08:02,169 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:02,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:05,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:05,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:06,329 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:06,610 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:07,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:07,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:10,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:10,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:11,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:08:11,548 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:08:11,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:08:11,701 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:12,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:12,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:12,331 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:14,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:14,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:16,715 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:16,851 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:16,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:18,333 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:19,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:19,634 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:20,263 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:08:21,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:21,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:22,299 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:22,334 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:24,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:24,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:26,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:26,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:26,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:08:26,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:08:26,548 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:08:27,879 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:28,336 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:29,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:29,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:31,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:31,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:33,617 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:34,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:34,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:34,337 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:36,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:36,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:36,697 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:08:36,699 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:08:36,699 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:08:36,700 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:08:37,338 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:08:38,339 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:38,702 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:39,148 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:08:39,164 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:39,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:41,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:41,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:43,703 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:44,305 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:08:44,306 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:08:44,307 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:08:44,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:44,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:46,342 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:08:47,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:47,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:49,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:49,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:50,425 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:08:50,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:52,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:52,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:55,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:55,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:55,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:08:57,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:57,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:59,516 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:08:59,516 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:08:59,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:08:59,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:08:59,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:00,721 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:01,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:01,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:04,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:04,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:05,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:06,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:06,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:08,346 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:09,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:09,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:11,087 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:11,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:11,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:12,347 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:14,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:14,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:14,348 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:14,387 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:09:14,388 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:09:14,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:09:16,349 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:16,384 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:17,752 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:09:17,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:17,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:18,349 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:20,267 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:09:20,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:20,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:21,686 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:22,351 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:22,860 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:22,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:24,351 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:25,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:25,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:26,352 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:26,987 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:27,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:27,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:29,387 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:09:29,388 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:09:29,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:09:30,353 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:30,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:30,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:32,301 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:32,353 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:32,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:32,952 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:34,354 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:35,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:35,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:37,628 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:38,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:38,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:38,355 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:40,356 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:41,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:41,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:42,357 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:42,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:43,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:43,080 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:44,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:09:44,388 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:09:44,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:09:45,975 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:45,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:46,358 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:48,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:48,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:48,257 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:48,360 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:50,268 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:09:50,360 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:51,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:51,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:53,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:53,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:53,565 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:54,362 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:56,362 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:56,815 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:09:56,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:56,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:58,363 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:09:58,882 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:09:59,387 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:09:59,388 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:09:59,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:09:59,699 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:09:59,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:01,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:01,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:02,364 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:04,199 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:04,365 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:04,631 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:04,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:06,366 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:06,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:06,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:09,523 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:09,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:09,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:10,367 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:11,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:11,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:12,368 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:13,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:13,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:13,869 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:10:13,870 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:10:13,870 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:10:13,871 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:10:14,369 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:10:14,370 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:14,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:10:14,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:10:14,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:10:14,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:16,370 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:16,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:16,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:18,371 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:18,592 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:18,602 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:20,162 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:20,269 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:10:20,372 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:21,625 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:21,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:22,373 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:23,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:23,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:25,482 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:26,375 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:26,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:26,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:28,376 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:28,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:28,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:29,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:10:29,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:10:29,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:10:30,376 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:30,806 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:31,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:31,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:34,378 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:35,186 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:10:35,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:35,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:36,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:36,379 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:38,380 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:38,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:38,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:40,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:40,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:41,433 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:42,381 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:43,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:43,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:44,382 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:44,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:10:44,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:10:44,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:10:45,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:45,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:46,383 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:46,754 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:48,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:48,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:50,272 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:10:50,385 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:50,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:50,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:52,071 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:52,385 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:53,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:53,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:54,386 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:55,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:55,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:57,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:10:57,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:57,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:10:59,387 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:10:59,388 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:10:59,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:10:59,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:11:00,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:00,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:01,388 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:02,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:02,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:02,715 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:03,389 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:05,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:05,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:07,390 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:07,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:07,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:08,027 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:09,390 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:09,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:09,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:11,391 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:12,659 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:11:12,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:12,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:13,348 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:14,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:11:14,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:11:14,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:11:15,392 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:15,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:15,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:17,393 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:17,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:17,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:18,665 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:19,394 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:20,273 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:11:20,344 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:11:20,346 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:11:20,346 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:11:20,346 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:11:20,394 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:11:20,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:20,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:22,395 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:22,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:22,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:23,395 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:23,980 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:24,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:24,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:25,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:27,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:27,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:27,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:29,307 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:29,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:11:29,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:11:29,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:29,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:29,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:11:31,397 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:31,517 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:31,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:33,398 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:34,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:34,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:34,642 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:35,398 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:36,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:36,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:39,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:39,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:39,399 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:39,965 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:41,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:41,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:41,400 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:43,401 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:44,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:44,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:44,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:11:44,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:11:44,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:11:45,609 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:46,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:46,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:47,402 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:49,402 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:49,922 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:11:49,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:49,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:50,275 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:11:51,276 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:51,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:52,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:52,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:53,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:54,953 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:54,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:56,910 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:11:57,405 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:57,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:57,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:11:59,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:11:59,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:11:59,405 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:11:59,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:11:59,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:00,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:01,406 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:02,238 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:02,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:02,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:04,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:04,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:05,407 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:07,408 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:07,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:07,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:07,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:09,413 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:09,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:09,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:11,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:11,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:12,874 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:13,415 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:14,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:12:14,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:12:14,528 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:12:14,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:14,883 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:15,416 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:16,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:16,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:17,417 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:18,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:19,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:19,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:20,276 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:12:21,418 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:21,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:21,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:23,419 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:23,509 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:24,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:24,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:25,420 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:26,849 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:12:26,850 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:12:26,851 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:12:26,851 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:12:27,265 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:12:27,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:27,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:27,420 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:12:28,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:28,852 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:29,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:12:29,389 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:12:29,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:29,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:12:30,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:30,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:31,422 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:32,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:32,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:33,423 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:34,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:35,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:35,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:37,424 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:37,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:37,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:39,425 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:39,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:40,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:40,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:41,425 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:42,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:42,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:44,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:12:44,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:12:44,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:12:44,599 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:45,427 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:45,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:45,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:47,427 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:47,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:47,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:49,429 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:50,116 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:50,278 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:12:50,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:50,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:52,751 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:52,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:53,430 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:55,431 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:55,441 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:12:55,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:55,665 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:57,431 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:12:57,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:57,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:12:59,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:12:59,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:12:59,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:13:00,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:00,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:00,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:01,433 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:02,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:02,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:03,434 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:05,435 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:06,077 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:06,102 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:13:06,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:06,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:09,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:09,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:09,436 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:11,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:11,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:11,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:11,437 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:13,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:13,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:13,437 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:14,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:13:14,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:13:14,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:13:16,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:16,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:16,714 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:17,439 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:18,050 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:18,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:19,439 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:20,279 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:13:21,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:21,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:21,441 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:22,280 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:23,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:23,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:25,442 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:26,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:26,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:27,359 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:27,443 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:28,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:28,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:29,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:13:29,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:13:29,443 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:29,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:13:31,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:31,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:32,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:33,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:33,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:33,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:13:33,363 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:13:33,363 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:13:33,364 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:13:33,445 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:13:33,445 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:34,445 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:35,445 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:36,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:36,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:37,446 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:37,996 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:38,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:38,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:41,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:41,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:41,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:43,318 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:43,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:44,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:13:44,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:13:44,529 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:13:44,687 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:13:44,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:44,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:45,449 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:46,738 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:46,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:48,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:49,450 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:49,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:49,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:50,280 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:13:51,451 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:51,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:51,607 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:53,452 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:53,945 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:54,477 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:54,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:56,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:56,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:57,453 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:59,269 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:13:59,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:13:59,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:13:59,454 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:13:59,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:13:59,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:13:59,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:01,454 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:01,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:01,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:04,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:04,588 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:04,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:05,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:06,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:06,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:07,457 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:09,457 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:09,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:09,635 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:09,913 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:11,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:11,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:13,460 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:14,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:14:14,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:14:14,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:14:14,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:14,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:15,461 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:15,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:16,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:16,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:17,461 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:19,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:19,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:20,282 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:14:21,212 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:21,463 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:23,410 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:14:23,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:23,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:23,464 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:25,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:25,465 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:25,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:26,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:28,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:28,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:29,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:14:29,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:14:29,466 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:29,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:14:30,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:30,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:31,466 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:31,603 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:33,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:33,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:33,467 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:35,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:35,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:37,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:37,468 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:38,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:38,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:39,469 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:39,870 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:14:39,871 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:14:39,872 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:14:39,872 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:14:40,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:40,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:40,470 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:14:41,470 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:42,470 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:42,506 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:43,450 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:43,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:43,471 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:44,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:14:44,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:14:44,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:14:45,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:45,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:47,472 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:47,834 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:48,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:48,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:49,473 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:50,284 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:14:50,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:50,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:51,474 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:53,156 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:53,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:53,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:55,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:55,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:55,475 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:57,476 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:58,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:58,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:14:58,471 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:14:59,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:14:59,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:14:59,477 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:14:59,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:15:01,941 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:15:01,960 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:01,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:03,478 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:03,785 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:05,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:05,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:05,479 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:07,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:07,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:07,480 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:09,102 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:10,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:10,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:11,482 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:12,134 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:12,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:13,482 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:14,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:15:14,391 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:15:14,519 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:14,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:15:15,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:15,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:15,483 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:17,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:17,102 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:19,485 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:19,751 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:20,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:20,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:20,285 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:15:21,485 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:22,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:22,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:23,486 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:25,060 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:25,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:25,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:27,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:27,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:27,487 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:29,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:15:29,390 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:15:29,489 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:29,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:15:30,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:30,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:30,369 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:31,489 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:32,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:32,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:34,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:34,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:35,490 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:35,680 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:36,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:36,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:37,491 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:39,491 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:40,473 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:15:40,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:40,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:41,006 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:43,493 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:43,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:43,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:44,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:15:44,391 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:15:44,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:15:45,493 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:45,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:45,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:46,326 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:46,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:15:46,354 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:15:46,355 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:15:46,355 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:15:46,494 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:15:47,494 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:48,494 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:48,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:48,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:50,287 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:15:50,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:50,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:51,495 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:51,647 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:53,496 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:53,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:53,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:55,497 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:55,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:55,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:56,974 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:15:58,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:58,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:15:59,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:15:59,391 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:15:59,498 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:15:59,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:16:00,595 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:00,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:01,499 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:02,302 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:03,500 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:03,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:03,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:05,653 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:05,666 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:07,501 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:07,625 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:08,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:08,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:09,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:10,593 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:10,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:11,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:12,958 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:13,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:13,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:14,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:16:14,391 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:16:14,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:16:15,505 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:15,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:15,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:17,506 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:18,278 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:19,057 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:16:19,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:19,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:19,506 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:20,289 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:16:21,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:21,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:23,508 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:23,604 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:23,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:24,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:25,509 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:26,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:27,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:27,510 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:28,921 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:29,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:29,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:29,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:16:29,391 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:16:29,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:16:31,511 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:32,019 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:32,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:33,511 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:34,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:34,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:34,242 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:35,512 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:36,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:36,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:38,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:39,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:39,514 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:39,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:41,514 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:41,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:42,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:43,515 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:44,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:44,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:44,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:16:44,391 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:16:44,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:16:44,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:46,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:46,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:47,517 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:48,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:49,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:49,518 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:50,189 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:50,290 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:16:51,518 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:52,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:52,126 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:52,878 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:16:52,879 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:16:52,880 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:16:52,880 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:16:53,519 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:16:54,148 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:54,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:54,519 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:55,505 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:16:55,520 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:57,476 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:16:57,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:57,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:16:57,520 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:59,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:16:59,391 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:16:59,521 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:16:59,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:17:00,341 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:00,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:00,512 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:02,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:02,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:03,522 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:05,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:05,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:05,523 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:06,151 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:07,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:07,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:07,524 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:10,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:10,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:11,466 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:11,525 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:12,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:12,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:13,526 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:14,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:17:14,391 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:17:14,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:17:15,527 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:15,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:15,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:16,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:17,578 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:17,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:19,528 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:20,292 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:17:20,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:20,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:21,529 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:22,116 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:22,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:22,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:23,530 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:25,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:25,555 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:27,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:27,531 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:27,573 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:27,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:29,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:17:29,392 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:17:29,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:17:29,532 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:30,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:30,656 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:31,533 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:32,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:32,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:32,757 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:35,535 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:36,018 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:17:36,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:36,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:37,535 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:38,080 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:38,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:38,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:39,536 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:40,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:41,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:41,537 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:43,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:43,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:43,948 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:44,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:17:44,392 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:17:44,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:17:45,538 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:45,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:45,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:47,539 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:48,740 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:48,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:48,992 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:49,539 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:50,294 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:17:51,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:51,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:53,540 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:53,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:53,911 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:54,070 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:55,541 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:55,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:55,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:57,542 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:17:59,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:59,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:17:59,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:17:59,392 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:17:59,420 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:17:59,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:17:59,585 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:17:59,586 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:17:59,586 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:17:59,587 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:18:00,543 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:18:00,543 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:01,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:01,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:01,544 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:03,544 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:03,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:03,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:04,718 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:05,545 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:05,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:05,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:09,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:09,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:09,546 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:10,046 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:11,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:11,059 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:11,547 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:13,549 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:14,391 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:18:14,392 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:18:14,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:18:14,579 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:18:14,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:14,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:15,382 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:17,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:17,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:17,549 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:19,549 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:19,550 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:19,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:20,295 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:18:20,710 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:21,551 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:22,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:22,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:24,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:24,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:25,552 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:26,036 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:26,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:26,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:27,552 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:29,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:29,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:29,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:18:29,392 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:18:29,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:18:29,553 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:31,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:31,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:31,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:33,554 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:34,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:34,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:35,555 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:36,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:36,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:36,682 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:37,555 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:39,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:39,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:41,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:41,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:41,556 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:42,005 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:43,557 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:44,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:18:44,392 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:18:44,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:18:44,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:44,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:45,558 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:46,604 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:46,614 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:47,086 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:49,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:49,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:49,559 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:18:50,297 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:18:52,298 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:52,726 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:18:52,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:52,756 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:55,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:55,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:57,299 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:18:57,403 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:57,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:18:59,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:18:59,393 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:18:59,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:19:00,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:00,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:02,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:02,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:02,382 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:03,565 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:05,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:05,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:05,565 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:07,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:07,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:07,566 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:07,697 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:10,313 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:10,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:11,567 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:12,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:12,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:13,010 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:13,568 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:14,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:19:14,393 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:19:14,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:19:15,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:15,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:15,568 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:17,350 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:19:17,351 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:19:17,352 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:19:17,352 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:19:17,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:17,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:17,569 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:19:18,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:18,569 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:19,570 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:20,299 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:19:20,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:20,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:21,570 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:22,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:22,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:23,571 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:23,634 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:25,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:25,400 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:27,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:27,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:27,571 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:28,946 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:29,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:19:29,392 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:19:29,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:19:29,573 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:30,795 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:19:30,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:30,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:31,573 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:33,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:33,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:34,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:35,575 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:35,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:35,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:37,575 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:38,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:38,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:39,569 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:39,576 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:40,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:40,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:43,577 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:43,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:43,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:44,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:19:44,393 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:19:44,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:19:45,562 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:45,578 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:45,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:45,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:47,578 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:48,806 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:48,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:50,301 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:19:50,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:50,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:51,302 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:51,579 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:53,580 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:53,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:53,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:55,581 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:19:55,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:55,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:56,504 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:19:58,816 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:58,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:19:59,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:19:59,393 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:19:59,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:19:59,582 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:00,849 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:00,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:01,582 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:01,819 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:03,584 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:03,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:03,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:05,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:05,812 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:07,130 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:07,585 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:09,172 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:20:09,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:09,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:09,585 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:11,586 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:12,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:12,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:12,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:14,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:14,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:14,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:20:14,393 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:20:14,532 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:20:15,587 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:16,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:17,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:17,588 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:17,743 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:19,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:19,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:19,589 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:20,302 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:20:21,277 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:22,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:23,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:23,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:23,747 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:20:23,748 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:20:23,749 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:20:23,749 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:20:24,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:24,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:24,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:20:24,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:25,590 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:27,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:27,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:27,591 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:28,385 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:29,392 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:20:29,393 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:20:29,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:29,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:29,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:20:31,592 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:32,487 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:32,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:33,593 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:33,700 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:34,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:34,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:35,594 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:37,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:37,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:39,016 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:39,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:39,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:39,594 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:41,595 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:42,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:42,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:43,596 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:44,336 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:44,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:20:44,393 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:20:44,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:20:44,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:44,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:45,596 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:48,155 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:20:48,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:48,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:49,598 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:49,658 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:50,305 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:20:51,082 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:51,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:51,598 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:53,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:53,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:53,599 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:54,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:20:56,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:56,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:57,600 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:20:58,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:58,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:20:59,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:20:59,394 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:20:59,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:20:59,602 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:00,572 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:01,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:01,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:01,602 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:03,367 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:03,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:05,604 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:05,613 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:06,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:06,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:07,605 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:08,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:08,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:09,605 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:10,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:10,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:10,936 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:13,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:13,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:13,606 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:14,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:21:14,393 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:21:14,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:21:15,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:15,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:15,607 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:16,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:17,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:17,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:17,608 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:19,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:19,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:20,306 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:21:21,609 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:22,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:22,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:22,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:23,610 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:25,354 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:21:25,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:25,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:25,611 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:27,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:28,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:28,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:29,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:21:29,394 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:21:29,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:21:29,612 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:30,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:30,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:21:30,247 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:21:30,247 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:21:30,247 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:21:30,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:30,612 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:21:31,615 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:32,616 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:32,866 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:33,130 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:33,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:33,616 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:35,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:35,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:37,617 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:38,167 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:38,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:38,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:39,618 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:40,213 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:40,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:41,618 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:43,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:43,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:43,483 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:44,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:21:44,394 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:21:44,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:21:45,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:45,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:45,619 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:47,620 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:48,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:48,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:48,812 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:49,620 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:50,308 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:21:50,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:50,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:53,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:53,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:53,622 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:54,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:55,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:55,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:55,622 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:57,624 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:21:58,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:58,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:21:59,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:21:59,394 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:21:59,526 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:21:59,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:22:00,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:00,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:01,625 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:03,625 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:03,800 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:22:03,819 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:03,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:04,766 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:05,626 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:06,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:06,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:08,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:08,774 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:09,628 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:10,082 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:11,629 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:11,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:11,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:13,630 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:13,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:13,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:14,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:22:14,394 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:22:14,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:22:15,393 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:16,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:16,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:17,631 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:18,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:18,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:19,632 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:20,309 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:22:20,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:21,632 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:21,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:21,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:23,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:23,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:25,633 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:26,026 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:26,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:26,848 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:27,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:28,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:28,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:29,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:22:29,394 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:22:29,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:22:29,634 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:30,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:30,916 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:31,343 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:33,636 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:33,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:33,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:35,636 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:35,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:35,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:36,670 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:36,696 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:22:36,698 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:22:36,698 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:22:36,699 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:22:37,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:22:37,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:38,637 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:38,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:38,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:41,638 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:41,992 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:42,333 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:22:42,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:42,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:43,639 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:44,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:44,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:44,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:22:44,394 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:22:44,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:22:45,640 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:47,313 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:47,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:47,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:49,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:49,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:49,641 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:50,311 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:22:51,642 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:52,505 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:52,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:52,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:53,643 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:54,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:54,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:57,424 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:57,436 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:57,645 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:22:57,938 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:22:59,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:22:59,394 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:22:59,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:59,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:22:59,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:22:59,645 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:01,646 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:02,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:02,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:03,252 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:04,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:04,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:05,647 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:07,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:07,606 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:07,648 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:08,576 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:09,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:09,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:09,649 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:12,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:12,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:13,649 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:13,890 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:14,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:23:14,395 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:23:14,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:23:14,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:14,589 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:15,650 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:16,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:16,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:17,650 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:19,202 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:19,651 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:19,796 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:23:19,815 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:19,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:20,313 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:23:22,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:22,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:23,652 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:24,527 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:24,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:24,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:25,653 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:27,654 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:27,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:27,868 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:29,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:23:29,395 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:23:29,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:23:29,572 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:29,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:29,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:31,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:31,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:31,940 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:33,655 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:34,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:34,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:35,161 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:35,656 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:36,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:36,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:39,658 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:39,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:39,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:40,477 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:41,658 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:41,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:41,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:43,158 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:23:43,160 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:23:43,160 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:23:43,160 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:23:43,659 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:23:43,660 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:43,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:43,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:44,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:23:44,395 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:23:44,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:23:44,660 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:45,591 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:46,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:46,642 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:47,660 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:48,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:48,674 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:49,661 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:50,314 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:23:51,103 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:51,558 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:51,569 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:51,661 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:53,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:53,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:55,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:56,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:23:57,220 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:23:57,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:57,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:23:57,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:23:59,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:23:59,395 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:23:59,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:23:59,663 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:00,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:00,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:01,578 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:02,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:02,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:03,665 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:04,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:04,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:05,666 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:06,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:06,909 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:07,037 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:07,666 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:08,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:08,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:11,667 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:12,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:12,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:12,354 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:13,668 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:14,115 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:14,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:14,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:24:14,395 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:24:14,534 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:24:15,669 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:17,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:17,171 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:17,673 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:19,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:19,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:19,670 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:20,315 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:24:21,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:21,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:21,671 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:22,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:23,671 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:23,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:23,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:26,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:26,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:27,673 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:28,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:28,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:28,295 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:29,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:24:29,395 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:24:29,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:24:29,673 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:30,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:30,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:31,674 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:33,608 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:34,462 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:24:34,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:34,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:35,675 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:36,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:36,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:37,676 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:38,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:39,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:39,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:39,677 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:41,657 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:41,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:43,679 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:44,254 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:44,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:24:44,395 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:24:44,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:24:44,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:44,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:45,679 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:46,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:46,618 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:47,680 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:49,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:49,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:49,566 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:49,592 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:24:49,594 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:24:49,594 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:24:49,595 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:24:49,680 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:24:50,317 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:24:50,681 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:51,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:51,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:51,681 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:53,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:54,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:54,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:54,877 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:24:55,682 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:24:56,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:56,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:58,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:58,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:24:59,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:24:59,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:24:59,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:24:59,684 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:00,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:01,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:01,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:01,684 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:03,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:03,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:03,685 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:05,492 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:06,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:06,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:07,686 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:08,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:08,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:09,687 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:10,809 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:11,688 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:11,992 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:25:12,011 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:12,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:14,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:25:14,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:25:14,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:25:14,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:14,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:15,689 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:16,120 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:16,942 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:16,951 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:17,690 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:18,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:18,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:19,690 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:20,319 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:25:21,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:21,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:21,747 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:23,691 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:23,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:23,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:25,692 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:26,757 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:26,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:26,800 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:27,692 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:28,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:28,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:29,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:25:29,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:25:29,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:25:31,694 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:31,970 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:31,980 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:32,084 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:33,694 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:34,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:34,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:35,695 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:36,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:36,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:37,390 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:38,696 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:38,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:39,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:41,927 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:41,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:42,697 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:42,706 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:43,956 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:43,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:44,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:25:44,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:25:44,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:25:44,697 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:45,989 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:45,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:46,698 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:48,021 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:49,328 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:25:49,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:49,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:50,320 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:25:50,700 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:52,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:52,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:52,701 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:53,347 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:54,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:54,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:54,701 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:56,027 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:25:56,028 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:25:56,029 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:25:56,029 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:25:56,702 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:25:56,702 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:57,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:57,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:58,657 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:25:58,703 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:25:59,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:59,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:25:59,395 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:25:59,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:25:59,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:26:00,703 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:02,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:02,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:02,704 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:03,972 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:04,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:04,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:06,705 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:07,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:07,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:08,706 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:09,290 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:09,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:09,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:10,706 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:12,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:12,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:14,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:14,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:14,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:26:14,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:26:14,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:26:14,564 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:14,707 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:16,708 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:17,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:17,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:18,708 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:19,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:19,331 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:19,930 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:20,321 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:26:22,344 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:22,354 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:22,710 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:24,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:24,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:24,711 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:25,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:26,711 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:27,882 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:26:27,904 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:27,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:29,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:26:29,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:26:29,535 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:26:30,572 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:30,713 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:30,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:30,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:32,714 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:32,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:32,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:34,715 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:35,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:35,887 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:35,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:37,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:37,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:38,716 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:40,717 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:40,906 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:40,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:41,224 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:42,718 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:42,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:42,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:44,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:26:44,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:26:44,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:26:46,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:46,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:46,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:46,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:48,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:48,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:48,719 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:50,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:50,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:50,323 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:26:50,720 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:51,854 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:52,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:52,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:54,721 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:54,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:54,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:56,721 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:57,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:57,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:57,169 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:26:58,722 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:26:59,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:26:59,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:26:59,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:26:59,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:26:59,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:01,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:01,834 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:02,489 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:02,515 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:27:02,516 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:27:02,517 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:27:02,517 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:27:02,724 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:27:02,724 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:04,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:05,341 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:27:05,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:05,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:06,725 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:07,805 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:08,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:08,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:10,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:10,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:10,726 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:12,727 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:13,126 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:13,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:13,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:14,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:27:14,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:27:14,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:27:14,728 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:15,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:15,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:18,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:18,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:18,434 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:18,729 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:20,325 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:27:20,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:20,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:20,729 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:22,730 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:23,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:23,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:23,746 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:25,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:25,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:26,731 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:28,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:28,296 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:28,732 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:29,066 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:29,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:27:29,396 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:27:29,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:27:30,314 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:30,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:30,733 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:33,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:33,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:34,379 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:34,734 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:35,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:35,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:36,734 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:38,264 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:38,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:38,735 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:39,692 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:40,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:40,310 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:42,736 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:43,816 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:27:43,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:43,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:44,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:27:44,397 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:27:44,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:27:44,737 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:45,008 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:46,737 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:46,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:46,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:48,791 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:48,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:50,323 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:50,326 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:27:50,738 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:51,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:51,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:52,739 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:53,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:53,761 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:54,739 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:55,639 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:27:56,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:56,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:58,741 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:27:58,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:58,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:27:59,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:27:59,397 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:27:59,536 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:28:00,741 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:00,950 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:01,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:01,850 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:02,742 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:03,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:03,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:04,742 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:06,258 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:06,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:06,824 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:08,744 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:08,842 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:08,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:08,948 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:28:08,950 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:28:08,950 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:28:08,950 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:28:09,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:28:10,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:11,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:11,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:11,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:11,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:12,745 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:13,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:13,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:14,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:28:14,397 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:28:14,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:28:16,746 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:16,899 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:16,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:16,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:18,747 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:18,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:18,964 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:20,328 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:28:20,747 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:22,222 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:22,302 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:28:22,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:22,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:24,749 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:25,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:25,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:26,750 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:27,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:27,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:27,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:28,750 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:29,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:28:29,397 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:28:29,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:28:30,129 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:30,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:32,166 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:32,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:32,751 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:32,852 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:34,752 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:35,154 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:35,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:36,752 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:37,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:37,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:38,165 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:40,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:40,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:40,753 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:42,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:42,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:42,754 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:43,481 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:44,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:28:44,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:28:44,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:28:44,754 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:45,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:45,245 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:47,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:47,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:48,756 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:48,793 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:49,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:49,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:50,330 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:28:50,757 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:28:51,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:51,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:54,332 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:54,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:54,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:56,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:56,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:59,332 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:28:59,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:28:59,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:28:59,482 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:28:59,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:59,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:28:59,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:29:02,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:02,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:04,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:04,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:04,760 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:04,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:06,761 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:07,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:07,417 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:09,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:09,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:10,144 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:10,762 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:12,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:12,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:12,762 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:14,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:29:14,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:29:14,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:29:14,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:14,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:14,763 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:15,563 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:17,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:17,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:18,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:19,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:19,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:20,332 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:29:20,765 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:21,333 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:22,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:22,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:22,766 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:24,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:24,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:26,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:26,736 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:29:26,738 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:29:26,738 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:29:26,739 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:29:26,767 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:29:26,767 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:27,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:27,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:28,767 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:29,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:29:29,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:29:29,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:29:29,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:29,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:29,768 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:30,768 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:32,027 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:32,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:32,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:34,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:34,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:34,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:36,770 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:37,337 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:38,191 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:29:38,210 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:38,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:38,771 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:41,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:41,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:42,643 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:42,772 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:43,032 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:43,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:44,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:29:44,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:29:44,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:29:44,773 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:46,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:46,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:46,774 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:47,950 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:48,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:48,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:50,334 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:29:50,775 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:51,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:51,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:52,775 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:53,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:53,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:53,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:54,776 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:56,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:56,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:58,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:58,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:29:58,572 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:29:58,777 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:29:59,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:29:59,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:29:59,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:30:00,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:00,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:00,778 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:02,778 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:02,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:03,002 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:03,871 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:05,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:05,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:06,780 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:07,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:07,069 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:08,780 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:09,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:09,869 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:09,879 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:10,781 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:11,897 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:11,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:14,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:30:14,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:30:14,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:30:14,626 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:14,782 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:15,439 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:30:15,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:15,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:16,782 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:18,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:18,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:18,783 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:19,789 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:20,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:20,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:20,336 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:30:20,784 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:23,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:23,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:24,785 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:25,107 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:25,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:25,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:26,786 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:28,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:28,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:28,786 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:29,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:30:29,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:30:29,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:30:30,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:30,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:30,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:32,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:32,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:32,787 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:33,096 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:30:33,097 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:30:33,097 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:30:33,098 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:30:33,788 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:30:34,788 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:35,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:35,215 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:35,727 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:35,789 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:36,789 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:37,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:37,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:40,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:40,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:40,790 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:41,036 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:42,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:42,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:42,791 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:44,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:30:44,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:30:44,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:30:44,791 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:45,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:45,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:46,351 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:47,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:47,230 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:48,793 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:50,253 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:50,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:50,337 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:30:50,793 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:51,660 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:52,794 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:53,772 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:30:53,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:53,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:55,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:55,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:56,795 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:56,969 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:30:58,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:58,639 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:30:58,796 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:30:59,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:30:59,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:30:59,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:31:00,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:00,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:00,797 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:02,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:03,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:03,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:04,798 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:05,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:05,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:06,799 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:07,578 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:08,707 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:08,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:08,799 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:10,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:10,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:12,800 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:12,882 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:13,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:13,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:14,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:31:14,398 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:31:14,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:31:14,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:15,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:15,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:16,801 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:18,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:18,854 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:18,865 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:20,339 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:31:20,803 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:20,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:20,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:22,803 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:23,501 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:24,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:24,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:24,804 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:26,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:26,086 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:28,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:28,812 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:29,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:29,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:29,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:31:29,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:31:29,538 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:31:30,805 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:32,525 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:31:32,547 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:32,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:32,806 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:34,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:34,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:34,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:36,810 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:37,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:37,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:38,811 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:39,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:39,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:39,455 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:31:39,457 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:31:39,457 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:31:39,457 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:31:39,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:39,811 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:31:40,811 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:41,812 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:42,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:42,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:44,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:31:44,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:31:44,452 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:44,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:44,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:31:44,612 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:44,812 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:46,813 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:47,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:47,363 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:48,814 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:49,383 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:49,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:50,047 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:50,340 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:31:52,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:52,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:52,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:54,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:54,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:54,816 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:55,361 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:31:56,817 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:31:57,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:57,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:59,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:31:59,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:31:59,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:59,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:31:59,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:32:00,556 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:00,819 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:02,373 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:02,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:02,819 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:04,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:04,419 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:04,820 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:05,979 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:07,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:07,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:08,821 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:10,822 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:11,120 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:32:11,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:11,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:11,288 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:12,822 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:14,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:14,201 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:14,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:32:14,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:32:14,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:32:14,823 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:16,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:16,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:16,573 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:18,824 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:19,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:19,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:20,342 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:32:20,825 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:21,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:21,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:21,908 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:22,825 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:24,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:24,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:26,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:26,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:26,827 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:27,209 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:28,827 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:29,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:32:29,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:32:29,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:29,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:29,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:32:30,828 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:31,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:31,460 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:32,509 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:34,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:34,385 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:34,829 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:36,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:36,418 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:36,830 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:37,815 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:38,830 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:39,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:39,453 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:41,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:41,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:42,831 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:43,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:43,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:43,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:44,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:32:44,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:32:44,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:32:44,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:45,801 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:32:45,803 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:32:45,803 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:32:45,803 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:32:45,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:32:46,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:46,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:46,832 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:47,833 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:48,430 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:49,856 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:32:49,876 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:49,889 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:50,343 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:32:50,834 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:51,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:51,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:52,834 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:53,741 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:54,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:54,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:54,835 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:56,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:56,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:58,836 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:32:59,043 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:32:59,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:32:59,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:32:59,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:32:59,901 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:32:59,912 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:00,837 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:01,931 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:01,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:02,837 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:04,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:04,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:05,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:06,838 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:07,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:07,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:08,839 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:09,661 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:10,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:10,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:10,840 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:12,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:12,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:14,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:33:14,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:33:14,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:33:14,841 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:14,971 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:15,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:15,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:16,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:17,079 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:17,088 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:18,842 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:20,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:20,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:20,272 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:20,345 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:33:22,108 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:22,118 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:22,843 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:24,844 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:25,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:25,131 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:25,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:26,845 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:28,642 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:33:28,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:28,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:29,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:33:29,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:33:29,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:33:30,618 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:30,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:30,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:30,846 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:32,847 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:33,464 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:33,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:34,847 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:35,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:35,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:36,196 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:38,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:38,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:38,849 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:40,577 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:40,587 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:40,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:41,507 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:42,850 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:43,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:43,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:44,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:33:44,399 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:33:44,539 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:33:45,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:45,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:46,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:46,851 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:48,617 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:48,630 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:48,852 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:50,346 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:33:50,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:50,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:50,853 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:52,123 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:52,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:33:52,151 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:33:52,151 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:33:52,152 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:33:52,853 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:33:53,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:53,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:53,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:54,854 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:55,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:55,734 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:56,855 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:57,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:33:58,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:58,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:33:58,855 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:33:59,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:33:59,400 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:33:59,540 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:34:00,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:00,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:00,856 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:02,758 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:03,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:03,684 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:04,857 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:06,857 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:07,265 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:34:07,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:07,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:08,063 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:08,858 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:09,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:10,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:12,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:12,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:12,859 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:13,364 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:14,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:34:14,400 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:34:14,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:34:14,860 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:15,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:15,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:16,860 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:17,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:17,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:18,680 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:20,348 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:34:20,470 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:20,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:20,862 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:22,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:22,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:22,862 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:23,987 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:24,863 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:25,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:25,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:27,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:27,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:28,864 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:29,288 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:29,399 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:34:29,400 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:34:29,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:34:30,605 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:30,615 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:30,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:32,633 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:32,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:32,865 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:34,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:35,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:35,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:36,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:37,560 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:37,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:38,867 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:39,912 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:40,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:40,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:40,868 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:42,646 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:42,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:44,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:34:44,400 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:34:44,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:34:44,869 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:45,220 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:46,097 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:34:46,117 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:46,127 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:46,870 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:48,870 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:49,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:49,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:50,349 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:34:50,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:51,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:51,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:52,872 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:54,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:54,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:54,873 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:55,835 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:34:56,283 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:56,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:56,873 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:34:58,517 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:34:58,518 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:34:58,519 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:34:58,519 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:34:58,873 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:34:59,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:59,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:34:59,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:34:59,400 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:34:59,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:34:59,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:00,874 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:01,150 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:01,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:01,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:02,875 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:04,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:04,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:04,876 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:06,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:06,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:06,457 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:08,878 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:09,441 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:09,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:10,878 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:11,481 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:11,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:11,768 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:12,879 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:14,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:14,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:35:14,400 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:35:14,406 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:14,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:35:16,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:16,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:16,880 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:17,079 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:18,881 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:19,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:19,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:20,351 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:35:20,881 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:21,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:21,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:22,352 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:24,882 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:24,910 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:35:24,929 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:24,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:26,883 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:27,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:28,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:28,027 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:28,884 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:29,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:35:29,400 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:35:29,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:35:30,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:30,060 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:32,890 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:32,998 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:33,044 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:33,061 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:34,891 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:35,087 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:35,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:36,892 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:38,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:38,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:38,310 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:40,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:40,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:40,893 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:42,894 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:43,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:43,187 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:43,617 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:44,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:35:44,401 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:35:44,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:35:44,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:45,207 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:45,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:46,895 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:47,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:47,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:48,931 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:49,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:50,007 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:50,352 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:35:50,897 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:52,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:52,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:52,897 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:54,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:54,111 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:54,237 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:35:54,898 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:56,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:56,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:58,899 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:35:58,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:58,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:35:59,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:35:59,401 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:35:59,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:35:59,617 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:00,900 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:02,433 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:36:02,454 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:02,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:02,901 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:04,849 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:04,876 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:36:04,878 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:36:04,878 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:36:04,878 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:36:04,902 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:36:05,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:05,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:05,902 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:06,902 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:07,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:07,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:08,903 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:10,149 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:10,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:10,567 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:10,903 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:12,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:12,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:14,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:36:14,401 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:36:14,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:36:14,905 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:15,452 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:15,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:15,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:16,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:17,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:17,526 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:18,906 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:20,354 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:36:20,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:20,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:20,756 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:22,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:22,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:22,907 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:24,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:24,490 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:24,909 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:26,063 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:26,910 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:27,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:27,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:29,272 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:29,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:29,400 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:36:29,401 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:36:29,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:36:30,911 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:31,373 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:32,294 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:32,304 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:32,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:34,323 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:34,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:34,912 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:36,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:37,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:37,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:38,913 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:40,866 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:36:40,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:40,895 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:40,914 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:41,986 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:42,914 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:42,915 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:42,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:44,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:36:44,401 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:36:44,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:36:45,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:45,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:46,916 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:47,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:47,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:47,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:48,917 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:50,356 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:36:50,917 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:50,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:50,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:52,357 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:52,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:52,958 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:54,919 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:56,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:56,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:56,919 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:57,906 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:36:58,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:58,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:36:58,920 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:36:59,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:36:59,401 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:36:59,543 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:37:01,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:01,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:02,922 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:03,110 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:03,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:03,216 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:04,922 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:05,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:05,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:06,923 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:08,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:08,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:08,518 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:10,120 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:10,132 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:10,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:11,194 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:37:11,196 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:37:11,196 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:37:11,197 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:37:11,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:37:12,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:12,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:12,925 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:13,826 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:13,926 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:14,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:37:14,401 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:37:14,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:37:14,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:14,917 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:14,926 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:18,511 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:37:18,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:18,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:18,927 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:19,136 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:20,357 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:37:20,570 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:20,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:20,928 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:22,929 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:23,386 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:23,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:24,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:25,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:25,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:26,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:27,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:27,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:28,930 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:29,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:37:29,401 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:37:29,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:37:29,624 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:30,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:30,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:30,932 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:32,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:32,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:32,933 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:35,058 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:35,261 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:35,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:36,934 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:37,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:37,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:38,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:39,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:39,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:40,362 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:40,935 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:42,096 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:42,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:44,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:44,137 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:44,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:37:44,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:37:44,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:37:44,937 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:45,593 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:46,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:46,174 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:46,937 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:48,938 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:48,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:48,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:50,359 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:37:50,982 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:50,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:51,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:52,939 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:54,463 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:37:54,482 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:54,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:54,940 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:56,289 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:37:56,941 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:37:57,419 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:57,432 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:59,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:37:59,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:37:59,457 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:59,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:37:59,544 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:38:00,942 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:01,581 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:02,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:02,583 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:02,943 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:04,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:04,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:04,944 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:06,915 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:07,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:07,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:08,945 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:09,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:09,746 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:10,946 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:12,225 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:12,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:12,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:12,947 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:14,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:38:14,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:38:14,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:38:14,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:14,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:16,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:16,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:16,948 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:17,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:17,564 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:38:17,565 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:38:17,565 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:38:17,566 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:38:17,948 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:38:18,948 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:19,694 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:19,704 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:19,949 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:20,361 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:38:20,949 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:21,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:21,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:22,847 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:24,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:24,661 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:24,950 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:26,680 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:26,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:26,951 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:28,154 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:28,952 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:29,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:38:29,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:38:29,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:38:29,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:29,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:32,953 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:33,119 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:38:33,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:33,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:33,467 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:34,953 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:35,173 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:35,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:36,954 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:38,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:38,188 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:38,783 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:40,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:40,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:40,955 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:42,956 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:43,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:43,276 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:44,097 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:44,401 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:38:44,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:38:44,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:38:44,956 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:45,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:45,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:48,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:48,176 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:48,958 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:49,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:50,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:50,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:50,362 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:38:50,958 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:38:52,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:52,892 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:54,364 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:54,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:54,919 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:57,375 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:57,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:59,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:38:59,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:38:59,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:59,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:38:59,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:38:59,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:39:02,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:02,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:04,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:04,114 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:04,964 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:05,206 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:06,965 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:06,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:06,998 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:08,966 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:10,500 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:10,519 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:39:10,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:10,551 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:12,967 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:13,685 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:13,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:14,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:39:14,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:39:14,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:39:14,968 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:15,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:15,723 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:15,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:16,969 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:18,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:18,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:20,364 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:39:20,796 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:20,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:20,970 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:21,365 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:22,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:22,841 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:22,971 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:24,972 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:25,706 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:25,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:26,421 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:27,741 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:27,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:28,973 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:29,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:39:29,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:39:29,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:39:30,616 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:30,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:30,974 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:31,722 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:32,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:32,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:32,975 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:35,050 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:39:35,051 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:39:35,051 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:39:35,052 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:39:35,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:35,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:35,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:39:35,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:36,976 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:37,053 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:37,719 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:37,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:38,977 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:39,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:39,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:40,978 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:42,328 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:42,659 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:42,671 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:44,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:39:44,402 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:39:44,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:39:44,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:44,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:44,979 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:46,980 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:47,633 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:48,183 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:39:48,202 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:48,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:48,980 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:50,366 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:39:51,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:51,133 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:52,937 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:52,982 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:53,153 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:53,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:54,982 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:56,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:56,161 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:56,983 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:39:58,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:58,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:39:58,243 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:39:59,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:39:59,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:39:59,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:40:00,985 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:01,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:01,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:02,986 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:03,218 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:03,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:03,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:04,986 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:05,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:05,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:07,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:08,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:08,853 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:08,987 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:10,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:10,034 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:10,988 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:12,989 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:13,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:13,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:14,153 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:14,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:40:14,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:40:14,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:40:15,063 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:15,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:16,991 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:18,091 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:18,103 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:18,991 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:19,453 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:20,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:20,143 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:20,367 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:40:20,992 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:22,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:22,183 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:24,761 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:24,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:25,428 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:40:25,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:25,459 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:26,994 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:28,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:28,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:28,995 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:29,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:40:29,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:40:29,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:40:30,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:30,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:30,598 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:33,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:33,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:33,998 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:35,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:35,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:35,999 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:36,027 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:37,999 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:38,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:38,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:40,000 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:40,437 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:40,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:41,342 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:41,369 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:40:41,370 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:40:41,370 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:40:41,371 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:40:42,001 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:40:43,461 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:43,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:44,002 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:44,402 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:40:44,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:40:44,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:40:45,493 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:45,503 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:46,002 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:46,616 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:48,003 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:48,497 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:48,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:50,369 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:40:50,528 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:50,537 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:51,953 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:52,004 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:53,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:53,543 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:54,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:55,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:55,575 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:56,005 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:40:57,267 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:40:58,624 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:58,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:40:59,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:40:59,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:40:59,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:41:00,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:00,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:00,670 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:02,007 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:02,574 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:04,008 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:04,267 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:41:04,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:04,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:07,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:07,247 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:07,881 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:08,009 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:09,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:09,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:10,011 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:12,011 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:12,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:12,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:13,191 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:14,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:14,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:14,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:41:14,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:41:14,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:41:16,012 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:16,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:16,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:18,013 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:18,501 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:18,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:18,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:20,014 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:20,370 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:41:20,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:20,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:23,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:23,737 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:23,806 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:24,015 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:25,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:25,773 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:26,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:28,016 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:28,795 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:28,809 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:29,111 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:29,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:41:29,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:41:29,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:41:30,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:30,837 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:32,018 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:33,856 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:33,867 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:34,018 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:34,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:35,887 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:35,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:36,019 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:37,922 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:37,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:39,713 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:40,021 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:41,161 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:41:41,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:41,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:42,021 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:44,022 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:44,292 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:44,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:44,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:41:44,404 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:41:44,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:41:45,017 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:46,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:46,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:47,691 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:41:47,692 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:41:47,693 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:41:47,693 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:41:48,023 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:41:48,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:49,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:49,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:50,024 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:50,313 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:50,372 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:41:51,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:51,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:52,025 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:54,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:54,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:55,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:41:56,027 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:56,184 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:56,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:58,027 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:41:58,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:58,232 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:41:59,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:41:59,403 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:41:59,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:42:00,028 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:00,921 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:00,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:00,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:03,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:03,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:04,030 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:05,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:06,001 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:06,031 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:06,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:08,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:08,031 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:08,032 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:11,123 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:11,141 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:11,542 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:12,033 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:13,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:13,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:14,034 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:14,403 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:42:14,404 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:42:14,546 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:42:16,035 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:16,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:16,112 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:16,587 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:19,670 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:42:19,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:19,702 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:20,036 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:20,373 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:42:21,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:21,732 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:22,037 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:22,155 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:24,038 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:24,675 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:24,691 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:26,038 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:26,713 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:26,724 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:27,458 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:29,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:42:29,404 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:42:29,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:42:29,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:29,736 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:30,040 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:31,758 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:31,768 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:32,041 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:32,760 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:34,042 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:34,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:34,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:36,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:36,858 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:38,043 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:38,064 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:38,884 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:38,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:40,044 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:41,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:41,749 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:42,044 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:43,371 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:43,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:43,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:44,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:42:44,404 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:42:44,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:42:46,046 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:46,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:46,672 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:48,047 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:48,685 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:48,697 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:48,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:50,047 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:50,375 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:42:50,731 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:50,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:53,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:53,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:53,996 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:54,022 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:42:54,024 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:42:54,024 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:42:54,024 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:42:54,049 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:42:54,049 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:56,049 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:57,149 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:42:57,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:57,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:42:58,050 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:42:59,313 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:42:59,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:42:59,405 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:42:59,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:43:00,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:00,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:02,051 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:02,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:02,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:04,052 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:04,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:05,345 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:05,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:06,053 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:07,377 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:07,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:09,412 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:09,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:09,925 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:10,054 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:12,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:12,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:12,191 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:14,055 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:14,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:14,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:14,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:43:14,405 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:43:14,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:43:15,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:16,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:16,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:18,056 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:19,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:19,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:20,057 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:20,377 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:43:20,379 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:21,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:21,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:22,058 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:24,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:24,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:25,841 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:26,059 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:26,157 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:26,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:28,060 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:29,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:29,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:29,404 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:43:29,405 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:43:29,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:43:30,060 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:31,142 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:31,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:31,236 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:34,062 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:34,741 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:43:34,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:34,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:36,062 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:36,449 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:37,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:37,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:38,063 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:39,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:39,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:41,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:41,647 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:41,754 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:42,064 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:44,065 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:44,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:43:44,405 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:43:44,515 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:44,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:44,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:43:46,066 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:46,546 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:46,557 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:47,059 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:49,579 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:49,591 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:50,067 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:50,378 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:43:51,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:51,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:52,068 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:52,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:54,069 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:54,735 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:54,759 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:56,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:56,786 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:57,675 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:43:58,070 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:43:59,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:43:59,405 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:43:59,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:43:59,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:43:59,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:00,070 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:00,358 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:44:00,360 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:44:00,360 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:44:00,360 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:44:01,071 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:44:01,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:01,838 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:02,071 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:02,988 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:04,722 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:04,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:06,073 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:06,752 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:06,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:08,073 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:08,294 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:09,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:09,788 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:10,074 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:12,075 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:13,341 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:44:13,361 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:13,372 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:13,603 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:14,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:44:14,405 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:44:14,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:44:15,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:15,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:16,076 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:18,077 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:18,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:18,469 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:18,910 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:20,077 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:20,379 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:44:20,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:20,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:23,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:23,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:24,078 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:24,217 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:25,545 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:25,554 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:26,079 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:28,080 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:28,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:28,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:29,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:44:29,406 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:44:29,547 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:44:29,603 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:30,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:30,478 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:32,081 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:32,498 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:32,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:34,082 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:34,825 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:35,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:35,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:36,083 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:37,402 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:37,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:40,084 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:40,128 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:40,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:40,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:42,085 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:42,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:42,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:44,085 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:44,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:44:44,406 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:44:44,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:44:45,258 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:45,270 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:45,431 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:47,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:47,301 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:48,087 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:50,088 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:50,380 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:44:50,714 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:44:50,742 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:50,743 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:50,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:52,090 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:53,475 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:53,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:55,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:55,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:56,057 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:44:56,091 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:58,091 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:44:58,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:58,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:44:59,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:44:59,406 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:44:59,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:45:00,092 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:00,584 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:00,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:01,368 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:03,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:03,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:04,094 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:05,506 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:05,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:06,095 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:06,676 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:06,703 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:45:06,704 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:45:06,705 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:45:06,705 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:45:07,095 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:45:08,095 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:08,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:08,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:10,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:10,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:11,981 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:12,097 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:13,640 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:13,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:14,098 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:14,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:45:14,406 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:45:14,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:45:15,676 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:15,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:16,098 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:17,291 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:18,698 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:18,709 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:20,100 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:20,382 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:45:20,729 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:20,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:22,100 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:22,384 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:23,658 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:23,668 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:24,101 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:25,692 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:25,701 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:27,909 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:28,102 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:29,120 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:45:29,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:29,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:29,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:45:29,406 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:45:29,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:45:30,103 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:31,976 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:31,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:32,104 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:33,213 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:34,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:34,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:36,105 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:37,180 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:37,192 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:38,106 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:38,521 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:39,211 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:39,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:40,107 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:42,246 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:42,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:43,823 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:44,108 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:44,278 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:44,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:44,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:45:44,406 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:45:44,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:45:46,109 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:47,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:47,306 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:48,110 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:49,132 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:49,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:49,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:50,384 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:45:52,111 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:52,467 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:52,483 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:54,112 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:54,439 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:45:54,508 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:54,518 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:56,112 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:57,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:57,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:58,113 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:45:59,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:45:59,406 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:45:59,471 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:59,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:45:59,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:45:59,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:02,114 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:02,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:02,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:04,115 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:04,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:04,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:05,047 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:06,116 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:07,847 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:46:07,871 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:07,880 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:10,117 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:10,353 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:10,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:10,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:12,118 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:12,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:12,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:13,032 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:46:13,033 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:46:13,033 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:46:13,033 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:46:13,118 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:46:14,119 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:14,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:46:14,406 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:46:14,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:46:15,633 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:15,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:15,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:17,899 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:17,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:18,121 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:20,121 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:20,385 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:46:20,966 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:21,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:21,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:22,122 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:23,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:23,122 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:26,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:26,052 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:26,124 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:26,270 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:28,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:28,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:28,125 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:29,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:46:29,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:46:29,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:46:30,125 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:30,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:30,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:31,576 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:32,990 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:32,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:34,126 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:36,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:36,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:36,127 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:36,874 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:38,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:38,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:38,128 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:41,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:41,200 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:42,129 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:42,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:43,219 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:43,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:44,130 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:44,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:46:44,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:46:44,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:46:46,131 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:46,601 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:46:46,619 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:46,628 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:47,491 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:49,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:49,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:50,132 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:50,387 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:46:51,462 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:51,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:52,133 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:52,796 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:53,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:53,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:54,134 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:56,320 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:56,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:58,097 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:46:58,135 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:46:58,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:58,374 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:46:59,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:46:59,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:46:59,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:47:00,136 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:01,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:01,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:02,137 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:03,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:03,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:03,401 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:06,139 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:06,445 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:06,455 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:08,139 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:08,474 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:08,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:08,698 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:10,140 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:11,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:11,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:13,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:13,416 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:14,008 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:14,141 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:14,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:47:14,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:47:14,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:47:16,142 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:16,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:16,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:18,143 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:18,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:18,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:19,310 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:19,336 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:47:19,337 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:47:19,338 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:47:19,338 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:47:20,144 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:47:20,389 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:47:21,144 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:21,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:21,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:22,145 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:24,145 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:24,612 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:25,207 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:47:25,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:25,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:26,146 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:27,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:27,282 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:29,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:47:29,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:47:29,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:47:29,917 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:30,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:30,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:30,147 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:32,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:32,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:32,148 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:34,149 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:34,939 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:34,949 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:35,223 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:36,150 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:36,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:36,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:39,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:39,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:40,151 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:40,533 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:42,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:42,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:42,152 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:44,152 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:44,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:47:44,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:47:44,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:47:44,993 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:45,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:45,541 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:47,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:47,035 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:48,154 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:50,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:50,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:50,155 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:50,391 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:47:51,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:52,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:52,083 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:52,155 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:55,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:55,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:56,157 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:56,461 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:47:57,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:57,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:47:58,157 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:47:59,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:47:59,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:47:59,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:48:00,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:00,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:00,158 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:01,597 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:03,845 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:48:03,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:03,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:04,159 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:06,160 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:06,910 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:06,924 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:07,081 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:08,161 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:08,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:08,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:11,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:12,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:12,162 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:12,394 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:14,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:14,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:14,163 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:14,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:48:14,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:48:14,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:48:16,164 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:16,965 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:16,981 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:17,709 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:19,013 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:19,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:20,165 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:20,392 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:48:22,166 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:22,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:22,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:23,014 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:24,167 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:24,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:24,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:25,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:48:25,690 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:48:25,690 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:48:25,691 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:48:26,168 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:48:27,168 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:27,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:27,262 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:28,169 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:28,316 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:29,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:29,290 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:29,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:48:29,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:48:29,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:48:30,169 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:32,170 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:32,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:32,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:33,625 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:34,317 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:34,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:36,171 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:37,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:37,519 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:38,171 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:38,939 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:39,538 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:39,548 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:40,172 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:43,048 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:48:43,067 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:43,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:44,173 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:44,245 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:44,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:48:44,407 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:48:44,549 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:48:45,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:45,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:46,174 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:48,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:48,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:48,175 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:49,312 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:50,175 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:48:50,394 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:48:50,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:50,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:52,821 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:52,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:54,396 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:55,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:55,318 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:57,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:57,351 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:48:59,397 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:48:59,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:48:59,408 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:48:59,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:49:00,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:00,049 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:02,071 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:02,081 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:04,600 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:04,969 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:04,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:06,181 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:07,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:07,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:08,182 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:09,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:09,897 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:09,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:10,182 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:11,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:11,938 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:14,184 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:14,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:49:14,408 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:49:14,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:49:14,977 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:14,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:15,599 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:16,185 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:17,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:17,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:18,185 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:20,389 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:49:20,396 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:49:20,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:20,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:21,398 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:22,186 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:23,349 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:23,358 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:24,187 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:25,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:25,388 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:26,188 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:26,813 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:28,328 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:28,338 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:29,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:49:29,408 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:49:29,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:49:30,192 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:30,365 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:30,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:32,122 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:32,192 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:33,240 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:33,252 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:34,193 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:35,271 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:35,287 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:36,194 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:37,422 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:38,288 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:38,299 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:40,195 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:40,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:40,330 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:42,196 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:42,727 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:43,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:43,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:43,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:49:43,408 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:49:43,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:49:43,409 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:49:44,197 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:49:44,197 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:44,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:49:44,408 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:49:44,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:49:45,197 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:45,398 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:45,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:47,427 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:47,443 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:48,034 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:48,198 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:50,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:50,212 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:50,222 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:50,397 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:49:52,199 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:52,241 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:52,257 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:53,334 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:55,281 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:55,293 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:56,201 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:58,202 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:49:58,638 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:49:58,822 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:49:58,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:58,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:49:59,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:49:59,408 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:49:59,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:50:00,203 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:00,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:00,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:03,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:03,664 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:03,934 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:04,209 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:05,686 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:05,696 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:06,210 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:08,210 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:08,683 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:08,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:09,232 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:10,721 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:10,730 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:12,212 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:13,727 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:13,740 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:14,213 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:14,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:50:14,408 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:50:14,530 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:14,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:50:15,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:15,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:16,214 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:18,776 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:18,790 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:19,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:20,215 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:20,399 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:50:20,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:20,818 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:22,216 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:23,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:23,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:24,217 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:25,147 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:25,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:25,762 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:28,219 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:28,632 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:28,644 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:29,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:50:29,408 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:50:29,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:50:30,219 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:30,630 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:30,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:30,673 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:32,220 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:32,693 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:32,711 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:35,750 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:35,916 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:50:35,936 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:35,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:36,221 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:38,222 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:38,874 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:38,888 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:40,223 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:40,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:40,921 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:41,053 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:43,937 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:43,947 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:44,224 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:44,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:50:44,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:50:44,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:50:45,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:45,979 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:46,225 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:46,596 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:48,226 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:48,891 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:48,902 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:49,693 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:50:49,694 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:50:49,694 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:50:49,695 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:50:50,226 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:50:50,401 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:50:50,920 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:50,934 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:51,227 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:52,227 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:52,320 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:53,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:53,957 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:54,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:55,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:55,991 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:56,229 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:50:57,625 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:50:58,882 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:58,894 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:50:59,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:50:59,408 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:50:59,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:51:00,231 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:00,913 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:00,925 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:02,232 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:02,926 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:03,905 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:03,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:04,232 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:05,935 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:05,945 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:08,228 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:08,233 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:08,955 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:08,966 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:10,234 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:10,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:10,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:12,235 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:13,531 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:14,236 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:14,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:51:14,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:51:14,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:51:14,581 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:51:14,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:14,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:17,472 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:17,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:18,237 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:18,831 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:19,502 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:19,511 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:20,238 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:20,402 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:51:22,239 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:22,530 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:22,540 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:24,138 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:24,559 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:24,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:26,240 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:27,608 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:27,621 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:28,241 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:29,408 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:51:29,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:51:29,550 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:51:29,615 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:29,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:29,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:32,242 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:32,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:32,718 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:34,243 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:34,750 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:34,755 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:34,764 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:36,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:37,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:37,789 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:38,245 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:39,807 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:39,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:40,055 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:42,246 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:42,864 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:42,878 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:44,247 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:44,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:51:44,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:51:44,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:51:44,898 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:44,908 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:45,357 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:46,248 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:47,915 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:47,926 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:49,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:49,959 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:50,249 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:50,404 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:51:50,405 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:52,250 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:53,434 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:51:53,456 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:53,465 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:54,251 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:55,963 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:51:55,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:51:55,991 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:51:55,991 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:51:55,991 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:51:56,252 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:51:56,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:56,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:57,252 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:58,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:51:58,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:58,442 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:51:59,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:51:59,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:51:59,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:52:00,253 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:01,277 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:01,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:01,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:02,254 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:03,532 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:03,542 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:06,255 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:06,449 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:06,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:06,584 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:08,256 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:08,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:08,499 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:10,257 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:11,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:11,504 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:11,890 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:13,524 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:13,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:14,258 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:14,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:52:14,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:52:14,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:52:16,259 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:16,568 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:16,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:17,198 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:18,259 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:18,599 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:18,609 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:20,406 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:52:21,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:21,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:22,261 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:22,407 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:23,643 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:23,655 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:24,261 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:26,262 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:26,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:26,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:27,807 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:28,682 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:28,695 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:29,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:52:29,409 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:52:29,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:52:30,263 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:32,124 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:52:32,144 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:32,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:32,264 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:33,111 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:34,264 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:35,097 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:35,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:37,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:37,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:38,266 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:38,406 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:40,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:40,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:40,266 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:42,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:42,072 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:42,267 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:43,716 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:44,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:52:44,410 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:52:44,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:52:45,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:45,220 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:46,268 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:47,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:47,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:48,269 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:49,018 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:50,185 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:50,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:50,270 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:50,407 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:52:52,214 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:52,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:52,270 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:54,322 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:52:55,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:55,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:56,272 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:57,267 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:57,279 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:58,273 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:52:59,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:59,308 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:52:59,409 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:52:59,410 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:52:59,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:52:59,564 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:00,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:02,078 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:02,089 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:02,302 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:53:02,303 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:53:02,303 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:53:02,304 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:53:03,274 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:53:04,107 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:04,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:04,275 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:04,933 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:05,275 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:06,140 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:06,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:06,275 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:08,276 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:09,610 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:53:09,629 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:09,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:10,241 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:12,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:12,494 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:12,507 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:14,278 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:14,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:53:14,410 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:53:14,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:14,539 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:14,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:53:15,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:16,279 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:16,561 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:16,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:19,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:19,390 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:20,280 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:20,409 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:53:20,854 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:21,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:21,421 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:22,282 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:23,440 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:23,451 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:24,282 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:26,159 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:26,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:26,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:28,284 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:28,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:28,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:29,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:53:29,410 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:53:29,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:53:30,285 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:31,462 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:31,492 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:31,501 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:32,285 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:33,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:33,535 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:36,287 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:36,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:36,480 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:36,772 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:38,288 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:38,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:38,510 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:40,288 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:41,431 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:41,444 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:42,077 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:43,466 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:43,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:44,290 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:44,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:53:44,410 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:53:44,551 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:53:46,291 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:46,804 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:53:46,830 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:46,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:47,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:48,292 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:49,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:49,978 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:50,410 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:53:52,000 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:52,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:52,293 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:52,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:54,294 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:55,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:55,175 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:56,295 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:53:57,195 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:57,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:53:57,990 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:53:59,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:53:59,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:53:59,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:54:00,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:00,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:00,296 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:02,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:02,204 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:02,297 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:03,299 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:04,298 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:05,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:05,209 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:07,229 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:07,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:08,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:08,631 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:08,641 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 01:54:08,642 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 01:54:08,642 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 01:54:08,643 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 01:54:09,299 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 01:54:10,106 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:10,119 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:10,300 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:11,300 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:12,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:12,147 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:13,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:14,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:54:14,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:54:14,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:54:14,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:14,832 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:16,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:16,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:19,382 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:19,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:19,585 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:20,412 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:54:21,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:21,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:22,303 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:24,750 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:54:24,769 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:24,778 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:25,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:26,304 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:27,533 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:27,544 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:29,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:54:29,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:54:29,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:54:29,566 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:29,574 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:30,548 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:32,306 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:32,327 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:32,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:34,357 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:34,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:36,286 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:37,146 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:37,160 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:38,307 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:39,178 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:39,190 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:41,894 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:41,932 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:41,946 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:43,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:43,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:44,309 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:44,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:54:44,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:54:44,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:54:46,839 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:46,852 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:47,619 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:48,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:48,886 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:50,310 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:50,414 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:54:52,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:52,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:53,416 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:54,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:54,066 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:54,311 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:54:56,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:56,855 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:58,662 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:54:58,872 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:58,881 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:54:59,410 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:54:59,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:54:59,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:55:00,314 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:02,112 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:55:02,139 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:02,150 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:04,597 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:04,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:04,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:06,315 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:07,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:07,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:09,868 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:10,008 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:10,018 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:12,038 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:12,047 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:12,316 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:14,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:55:14,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:55:14,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:55:14,941 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:14,950 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:15,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:16,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:16,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:18,318 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:19,760 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:19,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:20,416 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:55:21,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:21,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:21,802 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:22,319 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:24,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:24,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:26,775 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:26,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:26,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:28,320 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:29,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:55:29,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:55:29,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:55:29,626 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:29,636 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:31,654 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:31,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:32,602 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:34,322 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:34,520 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:34,531 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:36,553 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:36,563 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:37,936 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:39,786 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:55:39,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:39,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:40,323 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:42,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:42,529 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:43,537 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:44,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:55:44,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:55:44,550 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:44,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:55:44,562 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:46,328 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:47,329 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:47,340 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:48,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:49,359 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:49,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:50,418 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:55:51,329 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:52,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:52,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:54,414 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:54,420 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:55:54,423 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:57,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:57,228 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:57,330 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:55:59,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:59,263 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:55:59,411 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:55:59,411 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:55:59,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:55:59,558 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:02,028 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:02,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:03,332 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:04,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:04,074 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:05,049 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:06,297 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:06,309 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:09,334 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:09,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:09,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:10,623 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:11,439 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:11,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:14,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:14,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:14,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:56:14,412 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:56:14,552 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:56:15,335 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:15,627 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:17,696 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:56:17,715 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:17,728 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:19,336 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:20,419 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:56:20,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:20,840 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:21,420 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:22,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:22,873 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:25,338 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:25,771 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:25,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:26,450 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:27,803 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:27,820 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:29,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:56:29,412 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:56:29,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:56:30,705 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:30,716 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:31,340 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:31,593 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:32,739 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:32,748 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:35,586 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:35,596 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:36,704 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:37,341 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:37,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:37,622 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:40,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:40,415 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:42,327 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:42,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:42,448 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:43,343 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:44,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:56:44,412 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:56:44,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:56:45,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:45,305 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:47,324 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:47,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:47,344 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:48,018 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:50,289 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:50,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:50,420 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:56:52,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:52,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:53,346 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:53,589 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:55,701 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:56:55,720 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:55,733 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:58,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:58,446 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:56:59,223 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:56:59,350 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:56:59,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:56:59,412 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:56:59,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:57:00,463 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:00,473 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:03,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:03,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:04,821 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:05,352 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:05,428 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:05,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:08,307 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:08,316 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:10,333 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:10,342 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:10,451 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:11,354 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:13,221 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:13,233 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:14,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:57:14,412 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:57:14,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:57:15,250 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:15,259 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:15,355 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:15,578 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:18,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:18,021 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:20,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:20,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:20,421 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:57:21,356 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:21,423 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:22,077 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:22,627 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:25,113 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:25,124 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:27,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:27,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:27,193 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:27,358 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:29,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:57:29,413 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:57:29,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:57:30,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:30,095 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:32,846 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:33,359 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:33,545 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:57:33,565 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:33,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:35,600 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:35,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:38,339 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:38,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:38,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:39,361 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:40,371 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:40,387 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:43,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:43,336 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:43,362 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:43,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:44,412 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:57:44,413 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:57:44,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:57:45,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:45,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:48,244 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:48,256 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:49,364 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:49,652 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:50,273 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:50,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:50,423 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:57:53,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:53,023 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:55,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:55,054 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:55,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:57:55,365 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:57:58,025 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:58,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:57:59,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:57:59,413 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:57:59,553 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:58:00,055 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:00,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:00,576 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:01,367 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:02,831 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:02,843 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:04,866 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:04,875 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:06,435 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:07,369 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:07,650 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:07,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:11,137 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:58:11,155 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:11,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:11,370 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:12,041 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:14,165 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:14,177 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:14,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:58:14,413 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:58:14,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:58:16,196 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:16,205 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:17,371 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:17,689 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:19,003 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:19,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:20,425 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:58:21,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:21,048 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:23,373 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:23,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:23,817 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:23,828 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:25,847 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:25,857 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:28,652 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:28,663 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:29,037 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:29,374 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:29,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:58:29,413 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:58:29,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:58:30,681 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:30,690 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:33,541 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:33,552 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:34,614 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:35,376 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:35,571 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:35,580 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:38,484 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:38,496 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:40,253 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:40,516 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:40,525 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:41,377 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:43,268 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:43,280 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:44,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:58:44,413 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:58:44,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:58:45,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:45,311 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:45,378 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:45,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:48,685 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:58:48,708 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:48,717 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:50,427 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:58:51,381 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:51,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:51,407 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:51,428 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:53,425 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:53,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:56,186 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:56,197 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:57,175 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:58:57,382 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:58:58,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:58,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:58:59,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:58:59,413 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:58:59,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:59:01,100 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:01,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:02,821 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:03,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:03,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:03,384 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:05,963 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:05,973 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:07,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:08,009 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:08,350 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:09,385 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:10,890 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:10,900 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:12,918 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:12,928 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:13,387 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:13,949 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:14,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:59:14,413 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:59:14,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:59:15,944 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:15,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:17,974 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:17,987 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:19,388 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:19,567 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:20,428 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:59:20,753 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:20,767 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:22,784 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:22,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:25,236 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:25,390 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:26,013 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 01:59:26,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:26,042 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:28,766 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:28,777 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:29,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:59:29,414 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:59:29,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:59:30,606 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:30,799 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:30,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:31,392 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:33,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:33,576 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:35,594 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:35,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:36,417 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:37,393 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:38,458 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:38,468 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:40,486 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:40,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:41,394 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:42,008 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:43,394 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:43,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:44,413 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:59:44,414 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:59:44,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 01:59:45,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:45,434 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:47,396 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:47,646 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:48,326 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:48,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:50,355 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:50,366 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:50,430 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 01:59:53,214 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:53,255 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:53,269 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:53,398 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:55,286 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:55,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:58,159 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:58,170 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 01:59:58,811 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 01:59:59,400 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 01:59:59,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 01:59:59,414 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 01:59:59,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:00:00,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:00,199 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:03,628 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:00:03,648 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:03,662 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:04,389 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:05,402 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:06,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:06,370 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:08,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:08,405 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:09,403 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:09,959 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:11,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:11,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:13,368 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:13,378 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:14,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:00:14,414 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:00:14,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:00:15,405 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:15,573 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:16,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:16,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:18,426 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:18,435 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:20,431 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:00:21,350 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:21,360 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:21,406 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:21,436 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:23,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:23,389 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:26,216 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:26,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:26,769 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:27,408 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:28,243 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:28,254 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:29,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:00:29,414 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:00:29,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:00:31,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:31,039 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:32,366 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:33,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:33,068 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:33,410 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:35,833 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:35,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:37,411 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:37,861 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:37,870 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:37,926 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:41,080 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:00:41,099 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:41,109 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:43,413 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:43,524 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:44,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:44,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:44,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:00:44,414 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:00:44,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:00:46,158 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:46,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:49,145 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:49,198 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:49,208 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:49,415 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:50,432 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:00:51,226 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:51,235 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:54,104 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:54,116 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:54,762 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:00:55,417 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:00:56,136 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:56,145 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:58,167 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:58,182 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:00:59,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:00:59,414 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:00:59,554 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:01:00,631 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:00,983 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:00,995 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:01,418 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:03,015 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:03,024 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:05,419 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:05,972 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:05,988 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:06,177 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:08,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:08,017 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:10,058 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:10,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:11,421 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:11,825 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:12,743 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:12,757 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:14,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:01:14,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:01:14,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:01:14,775 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:14,785 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:17,423 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:17,497 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:18,119 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:01:18,142 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:18,151 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:20,434 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:01:20,896 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:20,907 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:22,930 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:22,943 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:23,035 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:23,424 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:25,754 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:25,765 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:27,783 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:27,792 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:28,594 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:29,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:01:29,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:01:29,426 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:29,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:01:30,572 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:30,582 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:32,603 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:32,612 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:33,427 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:34,202 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:35,384 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:35,396 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:37,413 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:37,429 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:39,429 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:39,766 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:40,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:40,284 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:42,303 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:42,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:44,335 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:44,347 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:44,414 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:01:44,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:01:44,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:01:45,430 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:45,575 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:46,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:47,016 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:49,037 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:49,046 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:50,435 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:01:51,432 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:51,437 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:51,770 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:51,779 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:55,319 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:01:55,337 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:55,346 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:56,545 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:01:57,434 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:01:58,298 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:58,312 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:01:59,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:01:59,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:01:59,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:02:00,334 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:00,343 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:01,435 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:02,229 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:03,356 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:03,376 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:05,393 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:05,404 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:07,437 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:07,826 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:08,239 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:08,249 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:10,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:10,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:13,223 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:13,234 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:13,418 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:13,438 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:14,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:02:14,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:02:14,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:02:15,251 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:15,260 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:18,128 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:18,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:19,188 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:19,440 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:20,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:20,168 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:20,437 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:02:23,093 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:23,105 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:24,704 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:25,125 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:25,138 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:25,441 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:27,984 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:27,999 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:29,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:02:29,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:02:29,442 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:29,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:02:30,020 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:30,029 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:30,600 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:33,381 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:02:33,399 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:33,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:35,444 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:35,905 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:36,238 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:36,248 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:38,266 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:38,275 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:41,348 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:41,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:41,446 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:41,508 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:43,380 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:43,395 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:44,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:02:44,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:02:44,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:02:46,149 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:46,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:47,178 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:47,448 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:48,181 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:48,194 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:50,439 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:02:50,994 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:51,005 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:52,700 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:53,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:53,040 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:53,450 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:55,798 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:55,808 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:57,451 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:02:57,825 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:57,835 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:02:58,296 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:02:59,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:02:59,415 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:02:59,555 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:03:00,714 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:00,725 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:02,745 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:02,755 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:03,453 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:03:03,888 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:05,564 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:05,581 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:07,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:07,610 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:09,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:09,454 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:03:10,941 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:03:10,962 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:10,971 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:13,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:13,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:14,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:03:14,416 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:03:14,556 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:03:14,636 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:15,456 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:03:15,829 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:15,844 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:18,601 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:18,613 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:20,440 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:03:20,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:20,641 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:20,651 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:21,458 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:03:23,485 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:23,495 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:25,459 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:03:25,514 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:25,527 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:26,387 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:28,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:28,391 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:28,500 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 02:03:28,502 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 02:03:28,502 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 02:03:28,503 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 02:03:29,415 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:03:29,416 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:03:29,465 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 02:03:29,466 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:03:29,560 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:03:30,409 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:30,422 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:31,569 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:32,982 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:33,162 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:36,012 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:36,043 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:37,181 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:37,467 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:03:38,062 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:39,051 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:41,070 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:42,010 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:42,182 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:44,033 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:44,041 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:44,710 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:03:44,808 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:03:44,809 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:03:46,433 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:46,703 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:47,964 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:50,037 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:03:50,442 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:03:51,045 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:51,056 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:53,076 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:53,443 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:54,075 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:56,092 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:56,101 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:58,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:03:58,611 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:58,620 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:03:59,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:03:59,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:03:59,679 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:04:00,440 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: partial_history 2024-03-28 02:04:00,440 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 02:04:00,442 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 02:04:00,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 02:04:00,442 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 02:04:00,442 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 02:04:00,443 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 02:04:00,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 02:04:00,443 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 02:04:00,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 02:04:00,443 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 02:04:00,443 DEBUG SenderThread:1893870 [sender.py:send():382] send: history 2024-03-28 02:04:00,443 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: summary_record 2024-03-28 02:04:00,444 INFO SenderThread:1893870 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-28 02:04:00,473 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/wandb-summary.json 2024-03-28 02:04:00,638 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:00,649 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:01,474 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:04:03,152 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:03,163 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:03,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:05,179 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:05,189 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:07,780 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:07,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:08,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:09,811 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:09,822 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:12,285 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:12,295 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:13,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:14,315 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:14,325 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:14,667 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:04:14,678 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:04:14,679 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:04:16,805 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:16,813 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:18,836 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:18,845 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:18,877 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:20,443 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:04:21,321 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:21,332 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:23,352 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:23,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:24,444 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:26,394 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:04:26,411 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:26,420 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:28,853 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:28,862 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:29,445 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:29,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:04:29,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:04:29,679 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:04:30,877 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:30,885 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:33,353 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:33,362 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:34,882 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:35,381 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:35,392 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:37,772 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:37,781 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:39,801 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:39,810 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:39,882 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:42,369 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:42,379 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:44,397 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:44,410 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:44,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:04:44,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:04:44,679 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:04:45,811 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:47,026 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:47,036 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:49,053 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:49,064 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:50,445 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:04:51,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:51,476 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:51,489 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:53,509 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:53,521 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:55,997 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:56,006 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:56,446 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:04:58,022 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:58,030 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:04:59,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:04:59,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:04:59,679 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:05:01,067 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:05:01,085 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:01,094 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:01,850 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:03,660 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:03,669 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:05,687 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:05,700 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:06,851 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:08,265 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:08,274 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:10,291 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:10,300 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:11,851 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:12,782 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:12,794 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:14,668 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:05:14,679 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:05:14,679 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:05:14,814 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:14,826 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:17,408 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:17,419 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:17,849 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:19,438 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:19,447 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:20,446 DEBUG SenderThread:1893870 [sender.py:send():382] send: stats 2024-03-28 02:05:21,954 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:21,968 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:23,448 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:23,986 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:23,996 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:26,479 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:26,488 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:28,507 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:28,512 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:28,522 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:29,499 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:05:31,004 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:31,065 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:33,084 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:33,508 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:34,135 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:34,923 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-28 02:05:34,923 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: stop_status 2024-03-28 02:05:34,924 DEBUG SenderThread:1893870 [sender.py:send_request():409] send_request: stop_status 2024-03-28 02:05:37,473 ERROR gpu :1893870 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-28 02:05:37,491 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:37,500 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:37,502 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log 2024-03-28 02:05:39,139 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:40,156 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:40,364 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:43,193 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:43,203 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:44,977 DEBUG HandlerThread:1893870 [handler.py:handle_request():146] handle_request: status_report 2024-03-28 02:05:45,225 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:45,237 ERROR gpu :1893870 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-28 02:05:45,504 INFO Thread-13 :1893870 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft/wandb/run-20240327_190418-lwtbcr8s/files/output.log