2024-03-27 14:10:33,215 INFO StreamThr :1483490 [internal.py:wandb_internal():86] W&B internal server running at pid: 1483490, started at: 2024-03-27 14:10:33.214417 2024-03-27 14:10:33,217 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status 2024-03-27 14:10:33,219 INFO WriterThread:1483490 [datastore.py:open_for_write():87] open: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/run-golaq7b9.wandb 2024-03-27 14:10:33,220 DEBUG SenderThread:1483490 [sender.py:send():382] send: header 2024-03-27 14:10:33,273 DEBUG SenderThread:1483490 [sender.py:send():382] send: run 2024-03-27 14:10:33,572 INFO SenderThread:1483490 [dir_watcher.py:__init__():211] watching files in: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files 2024-03-27 14:10:33,572 INFO SenderThread:1483490 [sender.py:_start_run_threads():1136] run started: golaq7b9 with start time 1711545033.213735 2024-03-27 14:10:33,578 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: check_version 2024-03-27 14:10:33,578 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: check_version 2024-03-27 14:10:33,605 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: run_start 2024-03-27 14:10:33,931 DEBUG HandlerThread:1483490 [system_info.py:__init__():27] System info init 2024-03-27 14:10:33,931 DEBUG HandlerThread:1483490 [system_info.py:__init__():42] System info init done 2024-03-27 14:10:33,931 INFO HandlerThread:1483490 [system_monitor.py:start():194] Starting system monitor 2024-03-27 14:10:33,931 INFO SystemMonitor:1483490 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-03-27 14:10:33,932 INFO HandlerThread:1483490 [system_monitor.py:probe():214] Collecting system info 2024-03-27 14:10:33,932 INFO SystemMonitor:1483490 [interfaces.py:start():190] Started cpu monitoring 2024-03-27 14:10:33,933 INFO SystemMonitor:1483490 [interfaces.py:start():190] Started disk monitoring 2024-03-27 14:10:33,934 INFO SystemMonitor:1483490 [interfaces.py:start():190] Started gpu monitoring 2024-03-27 14:10:33,936 INFO SystemMonitor:1483490 [interfaces.py:start():190] Started memory monitoring 2024-03-27 14:10:33,938 INFO SystemMonitor:1483490 [interfaces.py:start():190] Started network monitoring 2024-03-27 14:10:33,980 DEBUG HandlerThread:1483490 [system_info.py:probe():151] Probing system 2024-03-27 14:10:33,986 DEBUG HandlerThread:1483490 [system_info.py:_probe_git():136] Probing git 2024-03-27 14:10:33,995 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:33,996 DEBUG HandlerThread:1483490 [system_info.py:_probe_git():144] Probing git done 2024-03-27 14:10:33,996 DEBUG HandlerThread:1483490 [system_info.py:probe():199] Probing system done 2024-03-27 14:10:33,997 DEBUG HandlerThread:1483490 [system_monitor.py:probe():223] {'os': 'Linux-5.4.0-166-generic-x86_64-with-glibc2.29', 'python': '3.8.10', 'heartbeatAt': '2024-03-27T13:10:33.980515', 'startedAt': '2024-03-27T13:10:33.210069', 'docker': None, 'cuda': None, 'args': ('--model_name_or_path=distil-whisper/distil-large-v3', '--dataset_name=mozilla-foundation/common_voice_16_1', '--dataset_config_name=hi', '--language=hindi', '--train_split_name=train+validation', '--eval_split_name=test', '--max_steps=5000', '--output_dir=./', '--per_device_train_batch_size=32', '--per_device_eval_batch_size=32', '--logging_steps=25', '--learning_rate=1e-4', '--warmup_steps=500', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--preprocessing_num_workers=1', '--dataloader_num_workers=4', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--freeze_encoder', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': 'run_speech_recognition_seq2seq.py', 'codePathLocal': 'run_speech_recognition_seq2seq.py', 'codePath': 'run_speech_recognition_seq2seq.py', 'git': {'remote': 'https://huggingface.co/sanchit-gandhi/distil-large-v3-hi-ft-frozen-encoder', 'commit': 'e7946df277d73ac75c34c2017b01c6d39e0275cd'}, 'email': 'sanchit@huggingface.co', 'root': '/home/sanchit/distil-large-v3-hi-ft-frozen-encoder', 'host': 'hf-dgx-01', 'username': 'sanchit', 'executable': '/home/sanchit/hf/bin/python', 'cpu_count': 64, 'cpu_count_logical': 128, 'cpu_freq': {'current': 2202.863015625, 'min': 1500.0, 'max': 2250.0}, 'cpu_freq_per_core': [{'current': 3171.209, 'min': 1500.0, 'max': 2250.0}, {'current': 2364.325, 'min': 1500.0, 'max': 2250.0}, {'current': 1572.649, 'min': 1500.0, 'max': 2250.0}, {'current': 1659.311, 'min': 1500.0, 'max': 2250.0}, {'current': 1886.877, 'min': 1500.0, 'max': 2250.0}, {'current': 1977.114, 'min': 1500.0, 'max': 2250.0}, {'current': 2391.085, 'min': 1500.0, 'max': 2250.0}, {'current': 1715.56, 'min': 1500.0, 'max': 2250.0}, {'current': 1660.921, 'min': 1500.0, 'max': 2250.0}, {'current': 3284.409, 'min': 1500.0, 'max': 2250.0}, {'current': 1662.262, 'min': 1500.0, 'max': 2250.0}, {'current': 1664.285, 'min': 1500.0, 'max': 2250.0}, {'current': 2370.48, 'min': 1500.0, 'max': 2250.0}, {'current': 1659.366, 'min': 1500.0, 'max': 2250.0}, {'current': 3291.736, 'min': 1500.0, 'max': 2250.0}, {'current': 1664.415, 'min': 1500.0, 'max': 2250.0}, {'current': 2283.506, 'min': 1500.0, 'max': 2250.0}, {'current': 1713.973, 'min': 1500.0, 'max': 2250.0}, {'current': 1714.618, 'min': 1500.0, 'max': 2250.0}, {'current': 1714.445, 'min': 1500.0, 'max': 2250.0}, {'current': 3316.361, 'min': 1500.0, 'max': 2250.0}, {'current': 1645.973, 'min': 1500.0, 'max': 2250.0}, {'current': 1659.14, 'min': 1500.0, 'max': 2250.0}, {'current': 1660.739, 'min': 1500.0, 'max': 2250.0}, {'current': 3273.321, 'min': 1500.0, 'max': 2250.0}, {'current': 1658.755, 'min': 1500.0, 'max': 2250.0}, {'current': 1663.103, 'min': 1500.0, 'max': 2250.0}, {'current': 1661.275, 'min': 1500.0, 'max': 2250.0}, {'current': 3194.935, 'min': 1500.0, 'max': 2250.0}, {'current': 1730.989, 'min': 1500.0, 'max': 2250.0}, {'current': 1727.119, 'min': 1500.0, 'max': 2250.0}, {'current': 1697.351, 'min': 1500.0, 'max': 2250.0}, {'current': 3275.205, 'min': 1500.0, 'max': 2250.0}, {'current': 1665.717, 'min': 1500.0, 'max': 2250.0}, {'current': 1663.68, 'min': 1500.0, 'max': 2250.0}, {'current': 1669.771, 'min': 1500.0, 'max': 2250.0}, {'current': 3334.278, 'min': 1500.0, 'max': 2250.0}, {'current': 1635.286, 'min': 1500.0, 'max': 2250.0}, {'current': 1664.293, 'min': 1500.0, 'max': 2250.0}, {'current': 1664.902, 'min': 1500.0, 'max': 2250.0}, {'current': 1795.097, 'min': 1500.0, 'max': 2250.0}, {'current': 1795.841, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.869, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.608, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.143, 'min': 1500.0, 'max': 2250.0}, {'current': 1793.214, 'min': 1500.0, 'max': 2250.0}, {'current': 1795.704, 'min': 1500.0, 'max': 2250.0}, {'current': 1792.981, 'min': 1500.0, 'max': 2250.0}, {'current': 3272.364, 'min': 1500.0, 'max': 2250.0}, {'current': 1663.677, 'min': 1500.0, 'max': 2250.0}, {'current': 1654.916, 'min': 1500.0, 'max': 2250.0}, {'current': 1657.783, 'min': 1500.0, 'max': 2250.0}, {'current': 1791.15, 'min': 1500.0, 'max': 2250.0}, {'current': 1796.079, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.617, 'min': 1500.0, 'max': 2250.0}, {'current': 1794.699, 'min': 1500.0, 'max': 2250.0}, {'current': 1656.917, 'min': 1500.0, 'max': 2250.0}, {'current': 3275.536, 'min': 1500.0, 'max': 2250.0}, {'current': 1655.448, 'min': 1500.0, 'max': 2250.0}, {'current': 1656.093, 'min': 1500.0, 'max': 2250.0}, {'current': 1656.343, 'min': 1500.0, 'max': 2250.0}, {'current': 1657.244, 'min': 1500.0, 'max': 2250.0}, {'current': 1844.721, 'min': 1500.0, 'max': 2250.0}, {'current': 3312.551, 'min': 1500.0, 'max': 2250.0}, {'current': 3298.468, 'min': 1500.0, 'max': 2250.0}, {'current': 2342.129, 'min': 1500.0, 'max': 2250.0}, {'current': 2427.115, 'min': 1500.0, 'max': 2250.0}, {'current': 2390.813, 'min': 1500.0, 'max': 2250.0}, {'current': 2441.533, 'min': 1500.0, 'max': 2250.0}, {'current': 2391.094, 'min': 1500.0, 'max': 2250.0}, {'current': 2459.387, 'min': 1500.0, 'max': 2250.0}, {'current': 2458.036, 'min': 1500.0, 'max': 2250.0}, {'current': 2306.435, 'min': 1500.0, 'max': 2250.0}, {'current': 3298.811, 'min': 1500.0, 'max': 2250.0}, {'current': 2370.203, 'min': 1500.0, 'max': 2250.0}, {'current': 2387.939, 'min': 1500.0, 'max': 2250.0}, {'current': 1744.029, 'min': 1500.0, 'max': 2250.0}, {'current': 2438.926, 'min': 1500.0, 'max': 2250.0}, {'current': 3106.564, 'min': 1500.0, 'max': 2250.0}, {'current': 2415.235, 'min': 1500.0, 'max': 2250.0}, {'current': 3301.717, 'min': 1500.0, 'max': 2250.0}, {'current': 2364.152, 'min': 1500.0, 'max': 2250.0}, {'current': 2352.175, 'min': 1500.0, 'max': 2250.0}, {'current': 1650.287, 'min': 1500.0, 'max': 2250.0}, {'current': 3311.0, 'min': 1500.0, 'max': 2250.0}, {'current': 3305.609, 'min': 1500.0, 'max': 2250.0}, {'current': 1651.466, 'min': 1500.0, 'max': 2250.0}, {'current': 1653.023, 'min': 1500.0, 'max': 2250.0}, {'current': 3276.845, 'min': 1500.0, 'max': 2250.0}, {'current': 1653.413, 'min': 1500.0, 'max': 2250.0}, {'current': 3302.647, 'min': 1500.0, 'max': 2250.0}, {'current': 1653.312, 'min': 1500.0, 'max': 2250.0}, {'current': 3303.875, 'min': 1500.0, 'max': 2250.0}, {'current': 1620.896, 'min': 1500.0, 'max': 2250.0}, {'current': 1652.028, 'min': 1500.0, 'max': 2250.0}, {'current': 1653.199, 'min': 1500.0, 'max': 2250.0}, {'current': 3299.115, 'min': 1500.0, 'max': 2250.0}, {'current': 1651.729, 'min': 1500.0, 'max': 2250.0}, {'current': 1651.899, 'min': 1500.0, 'max': 2250.0}, {'current': 2278.731, 'min': 1500.0, 'max': 2250.0}, {'current': 3291.881, 'min': 1500.0, 'max': 2250.0}, {'current': 2372.068, 'min': 1500.0, 'max': 2250.0}, {'current': 1855.559, 'min': 1500.0, 'max': 2250.0}, {'current': 1654.361, 'min': 1500.0, 'max': 2250.0}, {'current': 1968.15, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.484, 'min': 1500.0, 'max': 2250.0}, {'current': 1715.246, 'min': 1500.0, 'max': 2250.0}, {'current': 2443.499, 'min': 1500.0, 'max': 2250.0}, {'current': 2139.871, 'min': 1500.0, 'max': 2250.0}, {'current': 2161.111, 'min': 1500.0, 'max': 2250.0}, {'current': 1718.071, 'min': 1500.0, 'max': 2250.0}, {'current': 2151.675, 'min': 1500.0, 'max': 2250.0}, {'current': 2978.053, 'min': 1500.0, 'max': 2250.0}, {'current': 1655.523, 'min': 1500.0, 'max': 2250.0}, {'current': 1657.328, 'min': 1500.0, 'max': 2250.0}, {'current': 1657.12, 'min': 1500.0, 'max': 2250.0}, {'current': 2168.709, 'min': 1500.0, 'max': 2250.0}, {'current': 2050.918, 'min': 1500.0, 'max': 2250.0}, {'current': 2050.632, 'min': 1500.0, 'max': 2250.0}, {'current': 1656.328, 'min': 1500.0, 'max': 2250.0}, {'current': 1670.254, 'min': 1500.0, 'max': 2250.0}, {'current': 3291.107, 'min': 1500.0, 'max': 2250.0}, {'current': 1660.234, 'min': 1500.0, 'max': 2250.0}, {'current': 1654.604, 'min': 1500.0, 'max': 2250.0}, {'current': 1657.069, 'min': 1500.0, 'max': 2250.0}, {'current': 1659.425, 'min': 1500.0, 'max': 2250.0}, {'current': 1850.263, 'min': 1500.0, 'max': 2250.0}, {'current': 3277.475, 'min': 1500.0, 'max': 2250.0}], 'disk': {'/': {'total': 1757.8785285949707, 'used': 1516.8665618896484}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 5, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA DGX Display', 'memory_total': 4294967296}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 503.5396919250488}} 2024-03-27 14:10:33,997 INFO HandlerThread:1483490 [system_monitor.py:probe():224] Finished collecting system info 2024-03-27 14:10:33,997 INFO HandlerThread:1483490 [system_monitor.py:probe():227] Publishing system info 2024-03-27 14:10:33,998 INFO HandlerThread:1483490 [system_monitor.py:probe():229] Finished publishing system info 2024-03-27 14:10:34,002 DEBUG SenderThread:1483490 [sender.py:send():382] send: files 2024-03-27 14:10:34,002 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-metadata.json with policy now 2024-03-27 14:10:34,006 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: python_packages 2024-03-27 14:10:34,007 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: python_packages 2024-03-27 14:10:34,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:10:34,007 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:34,009 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:10:34,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:10:34,206 DEBUG SenderThread:1483490 [sender.py:send():382] send: telemetry 2024-03-27 14:10:34,206 DEBUG SenderThread:1483490 [sender.py:send():382] send: config 2024-03-27 14:10:34,207 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 14:10:34,207 DEBUG SenderThread:1483490 [sender.py:send():382] send: telemetry 2024-03-27 14:10:34,207 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 14:10:34,207 WARNING SenderThread:1483490 [sender.py:send_metric():1354] Seen metric with glob (shouldn't happen) 2024-03-27 14:10:34,207 DEBUG SenderThread:1483490 [sender.py:send():382] send: telemetry 2024-03-27 14:10:34,513 INFO wandb-upload_0:1483490 [upload_job.py:push():131] Uploaded file /tmp/tmp17fwro16wandb/4j712kp4-wandb-metadata.json 2024-03-27 14:10:34,573 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:10:34,574 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/requirements.txt 2024-03-27 14:10:34,574 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-metadata.json 2024-03-27 14:10:36,573 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:10:36,889 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:36,903 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:38,922 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:38,934 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:39,213 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:10:41,526 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:41,539 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:43,556 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:43,568 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:44,214 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:10:46,027 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:46,039 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:48,060 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:48,072 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:48,577 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:10:49,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:10:49,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:10:49,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:10:49,218 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:10:50,578 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:10:51,064 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:51,076 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:52,579 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:10:53,103 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:53,112 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:54,365 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:10:54,579 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:10:55,975 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:55,991 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:56,580 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:10:58,009 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:58,019 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:10:58,581 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:10:59,586 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:00,048 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:00,060 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:00,581 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:02,582 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:02,870 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:02,882 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:04,006 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:11:04,007 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:11:04,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:11:04,583 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:04,906 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:04,917 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:05,207 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:05,583 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/config.yaml 2024-03-27 14:11:07,831 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:07,846 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:10,585 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:10,711 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:11,425 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:11:11,442 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:11,453 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:12,585 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:14,398 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:14,409 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:14,586 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:16,427 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:16,436 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:16,587 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:16,658 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:18,587 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:19,006 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:11:19,007 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:11:19,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:11:19,295 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:19,306 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:20,588 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:21,327 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:21,341 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:22,253 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:22,589 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:24,257 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:24,267 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:24,590 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:26,287 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:26,303 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:26,590 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:28,199 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:28,591 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:29,202 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:29,212 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:30,501 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:11:30,503 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 14:11:30,503 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 14:11:30,503 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 14:11:30,504 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 14:11:30,504 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:11:30,504 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:11:30,504 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:11:30,592 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:30,592 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_created():271] file/dir created: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:11:31,230 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:31,242 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:32,592 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:33,801 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:33,938 DEBUG SystemMonitor:1483490 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-03-27 14:11:33,940 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:11:34,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:11:34,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:11:34,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:11:34,236 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:34,246 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:34,593 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:36,266 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:36,279 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:36,594 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:38,594 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:39,119 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:39,129 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:39,744 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:40,595 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:40,595 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/config.yaml 2024-03-27 14:11:41,150 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:41,159 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:42,596 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:44,188 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:44,198 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:44,596 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:45,339 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:46,220 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:46,232 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:46,597 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:48,598 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:49,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:11:49,007 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:11:49,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:11:49,897 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:11:49,915 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:49,925 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:50,598 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:50,892 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:52,599 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:52,742 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:52,752 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:54,599 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:54,770 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:54,782 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:56,229 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:11:56,600 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:57,761 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:57,777 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:58,601 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:11:59,797 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:11:59,807 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:00,601 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:01,837 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:02,602 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:02,776 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:02,788 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:03,942 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:12:04,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:12:04,007 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:12:04,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:12:04,603 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:04,810 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:04,820 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:06,604 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:07,795 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:07,884 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:07,898 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:08,604 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:09,918 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:09,929 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:10,605 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:11,785 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:12:11,786 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:12:11,787 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:12:11,787 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:12:12,606 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:12,606 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:12:12,846 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:12,858 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:13,403 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:14,607 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:14,884 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:14,893 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:16,608 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:17,776 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:17,786 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:18,609 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:19,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:12:19,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:12:19,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:12:19,803 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:19,813 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:20,501 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:22,610 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:22,665 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:22,675 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:24,611 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:24,693 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:24,703 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:25,925 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:26,611 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:28,111 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:12:28,132 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:28,143 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:28,612 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:30,613 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:30,888 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:30,902 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:31,524 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:32,614 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:32,927 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:32,937 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:33,943 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:12:34,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:12:34,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:12:34,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:12:34,614 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:35,926 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:35,936 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:36,615 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:37,492 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:37,956 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:37,968 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:38,615 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:40,617 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:40,971 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:40,984 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:42,618 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:43,002 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:43,014 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:43,082 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:44,618 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:46,006 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:46,019 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:46,619 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:48,038 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:48,050 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:48,620 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:49,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:12:49,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:12:49,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:12:49,156 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:50,621 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:50,937 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:50,952 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:52,621 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:52,972 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:52,983 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:53,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:12:53,032 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:12:53,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:12:53,032 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:12:53,622 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:12:54,622 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:54,667 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:12:55,013 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:55,022 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:56,623 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:57,754 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:57,767 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:58,623 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:12:59,808 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:12:59,819 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:00,624 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:00,645 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:02,625 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:02,767 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:02,785 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:03,945 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:13:04,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:13:04,007 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:13:04,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:13:04,625 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:06,262 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:06,451 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:13:06,469 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:06,483 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:06,626 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:08,627 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:09,558 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:09,588 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:10,627 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:11,609 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:11,626 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:12,229 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:12,628 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:14,519 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:14,538 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:14,628 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:16,592 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:16,604 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:16,629 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:17,852 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:18,630 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:18,637 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:18,648 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:19,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:13:19,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:13:19,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:13:21,471 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:21,493 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:22,632 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:23,513 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:23,534 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:23,808 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:24,632 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:26,527 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:26,554 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:26,633 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:28,589 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:28,599 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:28,634 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:29,215 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:30,634 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:31,472 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:31,491 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:32,635 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:33,510 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:33,526 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:33,947 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:13:34,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:13:34,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:13:34,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:13:34,186 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:13:34,187 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:13:34,188 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:13:34,188 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:13:34,635 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:34,636 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:13:35,188 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:36,389 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:36,428 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:36,636 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:37,636 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:38,462 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:38,472 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:38,637 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:40,637 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:40,811 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:41,474 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:41,493 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:42,638 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:44,639 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:45,202 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:13:45,219 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:45,230 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:46,640 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:46,774 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:48,038 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:48,048 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:48,641 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:49,007 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:13:49,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:13:49,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:13:50,082 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:50,092 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:52,391 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:52,643 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:53,065 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:53,082 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:54,643 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:55,117 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:55,127 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:56,644 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:13:58,079 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:58,098 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:13:58,357 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:13:58,645 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:00,118 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:00,133 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:00,645 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:02,646 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:03,109 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:03,129 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:03,949 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:14:03,950 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:04,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:14:04,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:14:04,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:14:04,647 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:05,166 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:05,179 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:06,647 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:08,122 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:08,140 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:08,648 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:08,957 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:10,182 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:10,199 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:10,649 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:12,649 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:13,142 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:13,166 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:14,650 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:14,935 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:15,185 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:15,198 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:15,600 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:14:15,601 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:14:15,602 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:14:15,602 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:14:15,650 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:14:16,651 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:17,651 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:18,037 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:18,056 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:18,652 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:19,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:14:19,008 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:14:19,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:14:20,105 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:20,119 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:20,228 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:20,653 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:22,654 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:23,616 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:14:23,652 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:23,672 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:24,655 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:25,522 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:26,655 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:26,662 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:26,676 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:28,656 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:28,722 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:28,732 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:30,656 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:31,500 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:31,854 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:31,869 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:32,657 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:33,918 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:33,929 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:33,950 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:14:34,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:14:34,009 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:14:34,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:14:34,658 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:36,659 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:36,804 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:36,847 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:37,127 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:38,659 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:38,871 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:38,881 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:40,660 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:41,783 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:41,807 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:42,661 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:43,111 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:43,828 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:43,848 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:44,661 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:45,869 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:45,884 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:46,662 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:48,662 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:48,736 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:48,840 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:48,860 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:49,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:14:49,009 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:14:49,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:14:50,663 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:50,902 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:50,913 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:52,664 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:52,948 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:52,961 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:54,665 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:54,704 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:14:55,927 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:55,968 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:56,665 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:57,020 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:14:57,021 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:14:57,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:14:57,022 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:14:57,666 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:14:57,990 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:58,006 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:14:58,666 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:14:59,666 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:00,340 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:00,667 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:01,680 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:15:02,529 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:02,539 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:02,667 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:03,951 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:15:04,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:15:04,009 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:15:04,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:15:04,578 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:04,593 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:04,668 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:06,318 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:06,669 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:07,556 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:07,579 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:08,669 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:09,626 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:09,636 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:10,670 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:11,956 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:12,671 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:12,781 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:12,802 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:14,672 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:14,840 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:14,856 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:16,672 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:17,675 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:17,694 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:17,937 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:19,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:15:19,009 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:15:19,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:15:19,673 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:19,718 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:19,736 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:21,674 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:22,570 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:22,590 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:23,560 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:23,675 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:24,619 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:24,636 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:25,676 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:27,676 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:27,742 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:27,762 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:29,517 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:29,677 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:29,782 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:29,795 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:31,678 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:32,788 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:32,801 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:33,679 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:33,953 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:15:34,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:15:34,009 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:15:34,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:15:34,836 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:34,845 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:35,163 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:35,680 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:37,680 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:37,681 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:37,700 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:38,483 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:15:38,484 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:15:38,485 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:15:38,485 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:15:38,681 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:15:39,681 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:41,123 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:41,377 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:15:41,394 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:41,406 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:41,682 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:43,429 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:43,443 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:43,683 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:45,684 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:46,162 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:46,185 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:46,746 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:47,685 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:48,207 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:48,220 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:49,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:15:49,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:15:49,010 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:15:49,686 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:51,085 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:51,106 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:51,686 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:52,718 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:53,126 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:53,144 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:53,687 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:55,688 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:56,049 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:56,070 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:57,688 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:15:58,119 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:58,129 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:15:58,376 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:15:59,689 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:00,154 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:00,169 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:01,689 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:03,045 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:03,064 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:03,690 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:03,955 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:16:03,956 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:04,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:16:04,009 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:16:04,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:16:05,102 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:05,111 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:05,691 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:07,691 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:08,166 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:08,187 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:09,331 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:09,692 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:10,220 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:10,230 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:11,693 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:13,407 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:13,423 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:13,693 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:14,947 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:15,449 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:15,463 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:15,694 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:17,695 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:19,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:16:19,010 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:16:19,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:16:19,209 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:16:19,241 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:19,254 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:19,695 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:19,955 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:19,968 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:16:19,970 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:16:19,970 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:16:19,971 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:16:20,696 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:16:21,696 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:22,124 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:22,204 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:23,697 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:24,249 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:24,270 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:25,697 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:25,938 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:27,183 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:27,212 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:27,698 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:29,233 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:29,248 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:29,698 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:31,578 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:31,699 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:32,324 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:32,340 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:33,700 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:33,956 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:16:34,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:16:34,009 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:16:34,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:16:34,360 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:34,373 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:35,700 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:36,415 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:36,442 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:37,547 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:37,701 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:39,294 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:39,305 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:39,702 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:41,332 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:41,345 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:41,702 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:43,189 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:43,387 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:43,407 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:43,704 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:45,704 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:46,142 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:46,165 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:47,705 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:48,192 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:48,212 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:48,215 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:49,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:16:49,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:16:49,010 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:16:49,706 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:51,065 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:51,087 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:51,706 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:53,125 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:53,134 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:53,707 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:54,182 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:55,707 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:57,151 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:16:57,624 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:57,641 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:57,708 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:16:59,449 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:16:59,665 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:16:59,684 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:01,706 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:01,719 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:03,958 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:17:04,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:17:04,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:17:04,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:17:04,608 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:04,628 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:05,191 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:06,648 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:06,667 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:09,763 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:09,855 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:10,432 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:11,711 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:11,886 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:11,895 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:12,773 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:17:12,774 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:17:12,775 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:17:12,776 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:17:13,712 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:13,712 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:17:14,697 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:14,720 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:15,712 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:15,781 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:16,741 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:16,754 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:17,713 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:18,789 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:18,806 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:19,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:17:19,010 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:17:19,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:17:19,714 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:21,431 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:21,618 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:21,633 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:21,714 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:23,654 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:23,664 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:23,715 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:25,716 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:26,537 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:26,548 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:27,413 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:27,716 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:28,568 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:28,578 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:29,717 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:31,568 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:31,579 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:31,718 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:33,053 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:33,718 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:33,959 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:17:34,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:17:34,010 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:17:34,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:17:35,170 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:17:35,189 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:35,200 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:35,719 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:37,221 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:37,231 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:37,720 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:39,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:39,720 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:40,033 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:40,046 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:41,721 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:42,068 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:42,077 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:43,722 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:44,670 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:45,167 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:45,179 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:45,723 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:47,197 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:47,206 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:47,723 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:49,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:17:49,010 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:17:49,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:17:49,724 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:50,116 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:50,128 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:50,649 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:51,725 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:52,151 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:52,160 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:53,725 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:54,659 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:17:54,660 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:17:54,660 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:17:54,661 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:17:54,725 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:17:55,392 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:55,402 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:55,662 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:17:55,726 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:57,424 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:57,436 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:17:57,726 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:17:59,727 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:00,362 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:00,381 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:01,132 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:01,728 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:02,420 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:02,431 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:03,728 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:03,960 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:18:04,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:18:04,010 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:18:04,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:18:05,546 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:05,566 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:05,729 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:07,121 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:07,596 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:07,609 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:07,730 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:09,730 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:10,477 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:10,497 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:11,730 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:12,743 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:13,731 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:14,157 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:18:14,178 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:14,197 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:15,732 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:17,376 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:17,397 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:17,733 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:18,734 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:19,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:18:19,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:18:19,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:18:19,432 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:19,447 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:19,733 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:21,734 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:22,292 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:22,312 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:23,735 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:24,350 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:24,359 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:24,386 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:25,735 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:27,293 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:27,307 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:27,736 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:29,341 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:29,350 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:29,737 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:30,361 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:31,738 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:32,201 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:32,216 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:33,738 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:33,962 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:18:34,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:18:34,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:18:34,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:18:34,263 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:34,274 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:35,739 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:35,989 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:36,002 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:18:36,003 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:18:36,004 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:18:36,004 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:18:36,300 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:36,311 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:36,739 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:18:37,740 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:39,290 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:39,312 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:39,741 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:41,354 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:41,369 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:41,741 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:41,965 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:43,742 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:44,205 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:44,228 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:45,742 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:46,269 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:46,282 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:47,610 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:47,743 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:49,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:18:49,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:18:49,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:18:49,231 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:49,265 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:49,744 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:51,745 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:52,919 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:18:52,942 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:52,956 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:53,580 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:53,746 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:55,747 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:55,787 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:55,806 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:57,747 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:18:57,850 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:57,861 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:18:59,220 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:18:59,748 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:00,917 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:00,935 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:01,749 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:02,981 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:02,994 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:03,750 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:03,964 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:19:04,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:19:04,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:19:04,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:19:05,036 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:05,049 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:05,188 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:05,750 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:07,751 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:07,931 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:07,952 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:09,752 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:09,992 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:10,002 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:10,823 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:11,752 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:12,860 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:12,886 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:13,753 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:14,906 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:14,920 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:15,754 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:16,796 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:17,485 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:19:17,486 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:19:17,487 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:19:17,487 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:19:17,755 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:17,755 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:19:17,758 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:17,782 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:19,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:19:19,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:19:19,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:19:19,755 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:19,800 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:19,813 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:21,756 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:22,439 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:22,682 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:22,701 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:23,757 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:24,723 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:24,742 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:25,758 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:27,568 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:27,587 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:27,759 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:28,418 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:29,759 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:31,288 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:19:31,332 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:31,345 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:31,760 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:33,761 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:33,966 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:19:33,968 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:34,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:19:34,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:19:34,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:19:34,400 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:34,414 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:35,761 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:36,451 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:36,462 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:37,762 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:39,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:39,521 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:39,541 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:39,763 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:41,567 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:41,583 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:41,764 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:43,765 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:44,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:44,418 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:44,436 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:45,765 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:46,489 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:46,505 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:47,766 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:49,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:19:49,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:19:49,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:19:49,430 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:49,457 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:49,767 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:50,637 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:51,481 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:51,497 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:51,768 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:53,768 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:54,501 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:54,524 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:55,652 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:19:55,769 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:56,548 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:56,565 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:57,770 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:58,980 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:19:58,981 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:19:58,981 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:19:58,982 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:19:59,535 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:59,558 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:19:59,771 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:19:59,771 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:20:01,577 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:01,592 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:01,641 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:01,772 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:03,772 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:03,966 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:20:04,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:20:04,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:20:04,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:20:04,573 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:04,664 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:05,773 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:06,686 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:06,704 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:07,288 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:07,774 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:09,775 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:10,136 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:20:10,180 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:10,192 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:11,776 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:13,044 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:13,069 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:13,265 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:13,777 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:15,099 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:15,115 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:15,777 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:17,778 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:17,937 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:18,181 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:18,919 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:19,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:20:19,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:20:19,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:20:19,779 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:20,199 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:20,208 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:21,779 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:23,231 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:23,255 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:23,780 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:24,893 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:25,279 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:25,292 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:25,781 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:27,781 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:28,408 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:28,431 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:29,782 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:30,450 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:30,469 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:30,543 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:31,782 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:33,368 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:33,386 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:33,783 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:33,968 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:20:34,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:20:34,011 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:20:34,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:20:35,445 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:35,456 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:35,784 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:36,520 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:37,785 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:38,374 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:38,388 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:39,785 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:40,432 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:40,442 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:40,501 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:20:40,502 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:20:40,503 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:20:40,503 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:20:40,786 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:20:41,786 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:42,157 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:43,472 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:43,494 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:43,787 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:45,515 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:45,527 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:45,788 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:47,788 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:48,116 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:48,934 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:20:48,961 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:48,982 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:49,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:20:49,012 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:20:49,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:20:49,789 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:51,689 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:51,713 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:51,790 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:53,757 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:53,760 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:53,772 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:53,790 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:55,791 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:56,811 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:56,821 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:57,792 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:20:58,858 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:58,868 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:20:59,724 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:20:59,793 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:01,787 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:01,793 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:01,812 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:03,794 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:03,830 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:03,845 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:03,970 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:21:04,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:21:04,012 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:21:04,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:21:05,216 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:05,795 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:06,829 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:06,852 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:07,795 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:08,899 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:08,910 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:09,796 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:10,342 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:11,770 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:11,797 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:11,798 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:13,797 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:13,829 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:13,844 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:15,798 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:16,334 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:16,813 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:16,832 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:17,799 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:18,855 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:18,874 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:19,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:21:19,012 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:21:19,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:21:19,800 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:21,800 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:21,872 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:21,895 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:21,966 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:21,980 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:21:21,981 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:21:21,981 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:21:21,982 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:21:22,801 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:21:23,801 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:23,919 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:23,934 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:25,802 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:27,802 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:27,951 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:28,419 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:21:28,680 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:28,699 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:29,803 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:30,747 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:30,756 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:31,804 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:33,570 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:33,618 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:33,635 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:33,805 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:33,972 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:21:34,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:21:34,012 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:21:34,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:21:35,654 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:35,667 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:35,805 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:37,806 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:38,662 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:38,686 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:39,550 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:39,806 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:40,704 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:40,719 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:41,807 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:43,580 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:43,600 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:43,808 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:45,187 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:45,635 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:45,645 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:45,809 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:47,810 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:48,517 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:48,536 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:49,011 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:21:49,012 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:21:49,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:21:49,810 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:50,558 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:50,576 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:51,157 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:51,812 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:53,414 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:53,435 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:53,812 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:55,457 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:55,469 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:55,813 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:56,788 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:21:57,814 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:21:58,451 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:58,474 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:21:59,814 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:00,494 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:00,516 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:01,815 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:02,768 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:03,444 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:22:03,446 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:22:03,446 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:22:03,446 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:22:03,473 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:03,496 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:03,816 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:03,816 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:22:03,973 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:22:04,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:22:04,012 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:22:04,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:22:05,816 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:07,203 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:22:07,229 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:07,255 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:07,817 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:08,395 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:09,818 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:10,160 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:10,185 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:11,818 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:12,236 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:12,247 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:13,819 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:14,362 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:15,176 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:15,262 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:15,820 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:17,283 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:17,293 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:17,820 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:19,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:22:19,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:22:19,013 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:22:19,821 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:19,988 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:20,213 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:20,230 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:21,821 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:22,272 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:22,285 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:23,822 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:25,098 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:25,123 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:25,823 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:25,965 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:27,143 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:27,164 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:27,823 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:29,824 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:30,155 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:30,175 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:31,623 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:31,825 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:32,222 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:32,235 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:33,825 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:33,975 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:22:34,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:22:34,012 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:22:34,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:22:35,300 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:35,322 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:35,826 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:37,371 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:37,381 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:37,597 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:37,827 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:39,827 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:40,397 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:40,493 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:41,828 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:42,537 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:42,549 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:43,254 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:43,829 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:44,940 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:22:44,941 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:22:44,942 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:22:44,942 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:22:45,829 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:45,830 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:22:45,934 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:22:45,977 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:46,000 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:47,830 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:48,793 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:48,814 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:49,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:22:49,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:22:49,013 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:22:49,157 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:49,831 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:50,842 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:50,860 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:51,831 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:53,800 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:53,820 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:53,832 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:54,218 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:22:55,833 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:55,839 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:55,854 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:57,834 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:58,703 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:58,718 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:22:59,834 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:22:59,851 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:00,765 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:00,775 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:01,835 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:03,828 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:03,835 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:03,852 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:03,977 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:23:04,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:23:04,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:23:04,013 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:23:05,836 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:05,838 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:05,888 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:05,897 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:07,837 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:08,723 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:08,745 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:09,837 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:10,765 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:10,784 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:10,842 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:11,838 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:13,739 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:13,761 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:13,838 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:15,784 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:15,799 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:15,840 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:16,398 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:17,840 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:18,911 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:18,938 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:19,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:23:19,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:23:19,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:23:20,965 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:20,988 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:22,156 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:24,489 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:23:24,520 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:24,533 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:27,157 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:27,380 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:27,405 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:29,447 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:29,462 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:29,843 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:31,844 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:32,410 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:32,429 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:32,635 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:33,844 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:33,978 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:23:34,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:23:34,013 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:23:34,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:23:34,446 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:34,462 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:35,845 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:37,298 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:37,317 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:37,846 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:38,054 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:38,068 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:23:38,069 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:23:38,069 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:23:38,070 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:23:38,846 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:23:39,338 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:39,352 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:39,846 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:41,847 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:42,337 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:42,366 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:43,848 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:44,044 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:44,391 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:44,413 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:45,848 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:47,356 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:47,375 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:47,849 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:49,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:23:49,013 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:23:49,028 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:23:49,398 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:49,415 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:49,850 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:51,241 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:51,850 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:52,459 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:52,473 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:53,851 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:54,627 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:54,643 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:55,851 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:56,333 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:23:57,655 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:57,680 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:57,852 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:23:59,701 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:59,714 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:23:59,853 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:01,853 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:02,311 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:03,831 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:24:03,854 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:03,980 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:24:04,012 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:24:04,013 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:24:04,028 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:24:04,305 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:04,327 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:05,855 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:06,359 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:06,384 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:07,855 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:07,955 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:08,438 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:08,448 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:09,856 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:11,265 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:11,299 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:11,857 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:13,340 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:13,350 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:13,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:13,857 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:15,858 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:16,283 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:16,306 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:17,858 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:18,333 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:18,349 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:19,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:24:19,013 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:24:19,028 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:24:19,493 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:24:19,859 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:20,734 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:24:20,734 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:20,734 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:24:20,735 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:24:20,859 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:24:21,304 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:21,324 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:21,859 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:23,357 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:23,367 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:23,860 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:25,860 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:26,077 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:26,190 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:26,221 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:27,861 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:28,242 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:28,256 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:29,861 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:31,242 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:31,267 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:31,862 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:32,062 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:33,294 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:33,309 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:33,863 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:33,982 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:24:34,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:24:34,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:24:34,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:24:35,863 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:36,162 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:36,178 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:37,709 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:37,864 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:38,221 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:38,233 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:39,864 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:41,627 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:24:41,658 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:41,675 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:41,865 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:43,676 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:43,866 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:44,405 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:44,428 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:45,867 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:46,448 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:46,470 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:47,869 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:48,492 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:48,510 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:48,702 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:49,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:24:49,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:24:49,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:24:49,870 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:51,255 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:51,279 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:51,870 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:53,305 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:53,326 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:53,871 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:54,685 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:24:55,356 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:55,372 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:55,871 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:57,872 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:24:58,176 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:58,196 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:24:59,873 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:00,218 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:00,235 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:00,339 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:00,352 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:25:00,353 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:25:00,354 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:25:00,354 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:25:00,873 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:25:01,874 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:03,181 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:03,204 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:03,874 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:03,984 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:25:04,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:25:04,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:25:04,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:25:05,225 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:05,239 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:05,875 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:06,321 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:07,876 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:08,216 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:08,234 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:09,876 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:10,278 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:10,287 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:11,877 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:11,939 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:13,091 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:13,112 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:13,877 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:15,132 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:15,148 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:15,878 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:17,879 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:17,923 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:19,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:25:19,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:25:19,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:25:19,500 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:25:19,767 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:19,785 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:19,879 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:21,826 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:21,836 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:21,880 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:23,566 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:23,881 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:24,918 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:24,940 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:25,881 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:26,984 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:26,998 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:27,882 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:29,534 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:29,883 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:29,948 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:29,967 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:31,883 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:31,992 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:32,010 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:33,884 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:33,985 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:25:34,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:25:34,013 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:25:34,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:25:34,954 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:34,973 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:35,166 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:35,885 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:37,020 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:37,032 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:37,885 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:39,886 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:39,945 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:39,967 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:40,173 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:41,830 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:25:41,832 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:25:41,832 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:25:41,833 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:25:41,887 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:41,887 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:25:42,013 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:42,023 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:43,888 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:45,096 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:45,117 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:45,888 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:46,138 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:47,137 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:47,146 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:47,889 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:49,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:25:49,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:25:49,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:25:49,890 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:50,010 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:50,030 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:51,778 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:51,890 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:52,056 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:52,075 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:53,891 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:55,214 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:55,234 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:55,892 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:57,754 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:25:57,893 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:25:58,850 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:25:58,868 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:58,891 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:25:59,893 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:01,796 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:01,815 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:01,894 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:03,401 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:03,839 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:03,858 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:03,895 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:03,987 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:26:04,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:26:04,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:26:04,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:26:05,895 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:06,831 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:06,851 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:07,896 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:08,870 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:08,885 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:09,389 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:09,897 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:11,889 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:11,897 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:11,910 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:13,898 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:13,946 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:13,955 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:15,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:15,899 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:16,816 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:16,838 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:17,900 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:18,862 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:18,877 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:19,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:26:19,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:26:19,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:26:19,900 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:21,000 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:21,720 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:21,742 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:21,901 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:23,325 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:26:23,326 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:26:23,327 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:26:23,327 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:26:23,764 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:23,777 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:23,902 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:23,902 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:26:25,902 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:26,639 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:26,646 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:26,665 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:27,902 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:28,705 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:28,721 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:29,903 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:31,682 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:31,702 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:31,904 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:32,622 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:33,731 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:33,755 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:33,905 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:33,989 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:26:34,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:26:34,014 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:26:34,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:26:35,905 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:37,395 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:26:37,430 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:37,443 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:37,906 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:38,262 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:39,906 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:40,166 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:40,177 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:41,907 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:42,213 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:42,223 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:43,908 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:44,235 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:45,308 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:45,333 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:45,908 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:47,351 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:47,365 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:47,909 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:49,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:26:49,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:26:49,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:26:49,910 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:50,312 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:50,339 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:50,770 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:51,910 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:52,360 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:52,382 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:53,911 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:55,225 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:55,246 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:55,848 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:26:55,912 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:57,264 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:57,280 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:26:57,912 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:26:59,913 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:00,247 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:00,270 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:01,481 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:01,913 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:02,290 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:02,307 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:03,914 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:03,990 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:27:04,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:27:04,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:27:04,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:27:04,333 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:04,350 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:04,841 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:27:04,842 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:27:04,842 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:27:04,843 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:27:04,914 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:27:05,915 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:06,486 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:07,214 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:07,236 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:07,915 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:09,268 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:09,279 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:09,916 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:11,916 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:12,136 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:12,154 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:12,460 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:13,917 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:15,835 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:27:15,856 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:15,872 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:15,918 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:17,918 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:18,108 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:18,895 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:18,916 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:19,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:27:19,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:27:19,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:27:19,919 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:20,936 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:20,949 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:21,920 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:23,921 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:23,942 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:23,963 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:24,104 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:25,922 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:26,007 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:26,016 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:27,922 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:28,991 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:29,011 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:29,727 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:29,923 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:31,051 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:31,060 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:31,924 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:33,925 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:33,991 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:27:34,005 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:34,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:27:34,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:27:34,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:27:34,031 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:35,714 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:35,926 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:36,061 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:36,085 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:37,926 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:39,013 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:39,032 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:39,927 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:41,052 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:41,067 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:41,324 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:41,932 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:43,929 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:43,953 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:43,978 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:45,930 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:46,001 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:46,016 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:46,321 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:27:46,323 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:27:46,323 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:27:46,324 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:27:46,324 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:46,930 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:27:47,931 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:49,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:27:49,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:27:49,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:27:49,156 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:49,172 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:49,932 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:51,199 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:51,216 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:51,932 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:52,284 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:53,933 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:54,999 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:27:55,019 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:55,032 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:55,933 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:57,934 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:27:57,936 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:27:58,102 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:58,122 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:27:59,935 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:00,143 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:00,158 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:01,940 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:03,206 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:03,225 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:03,919 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:03,937 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:03,993 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:28:04,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:28:04,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:28:04,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:28:05,264 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:05,275 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:05,938 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:07,939 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:08,410 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:08,438 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:09,291 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:09,940 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:10,467 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:10,488 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:11,941 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:12,509 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:12,523 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:13,941 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:15,272 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:15,668 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:15,694 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:15,942 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:17,715 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:17,732 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:17,942 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:19,015 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:28:19,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:28:19,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:28:19,757 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:19,773 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:19,943 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:20,921 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:21,944 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:22,813 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:22,835 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:23,945 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:25,318 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:25,332 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:25,945 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:26,921 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:27,373 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:27,505 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:27,605 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:28:27,607 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:28:27,607 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:28:27,608 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:28:27,946 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:27,946 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:28:29,947 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:30,709 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:30,730 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:31,930 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:31,948 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:33,948 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:34,015 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:28:34,048 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:28:34,048 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:28:34,049 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:28:36,950 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:36,954 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:37,102 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:28:37,121 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:37,132 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:38,950 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:39,775 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:39,794 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:40,951 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:42,582 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:42,595 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:42,604 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:42,952 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:44,891 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:44,910 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:44,952 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:46,953 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:47,199 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:47,220 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:48,585 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:48,954 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:49,015 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:28:49,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:28:49,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:28:49,255 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:49,268 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:51,319 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:51,344 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:52,955 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:54,239 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:28:54,459 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:54,479 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:54,956 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:56,523 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:56,533 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:56,957 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:58,580 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:28:58,957 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:28:59,877 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:00,221 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:00,958 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:01,895 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:01,909 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:02,959 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:03,951 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:03,961 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:03,996 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:29:04,015 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:29:04,016 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:29:04,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:29:04,960 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:05,880 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:06,800 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:06,945 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:06,960 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:08,961 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:08,965 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:09,358 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:29:09,360 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:29:09,360 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:29:09,360 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:29:09,962 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:09,962 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:29:10,071 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:10,962 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:11,026 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:12,090 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:12,327 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:12,963 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:14,964 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:16,597 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:29:16,842 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:16,854 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:17,004 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:17,965 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:18,878 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:19,015 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:29:19,205 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:29:19,205 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:29:19,965 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:20,106 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:21,966 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:22,134 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:22,144 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:22,656 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:23,967 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:24,169 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:24,180 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:25,967 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:27,274 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:27,286 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:27,968 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:28,645 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:29,308 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:29,320 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:29,969 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:31,346 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:31,365 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:31,970 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:33,970 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:33,998 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:29:33,998 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:34,015 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:29:34,015 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:29:34,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:29:34,471 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:34,490 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:35,971 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:36,511 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:36,523 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:37,972 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:38,544 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:38,555 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:39,895 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:41,397 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:41,409 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:43,430 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:43,447 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:44,896 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:46,481 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:46,492 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:48,512 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:48,615 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:49,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:29:49,017 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:29:49,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:29:50,715 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:51,679 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:51,689 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:51,976 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:53,976 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:55,536 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:29:55,557 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:55,566 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:55,977 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:56,122 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:29:57,586 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:57,597 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:29:57,978 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:29:59,978 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:00,580 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:00,591 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:01,979 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:02,116 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:02,611 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:02,622 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:02,806 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:30:02,807 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:30:02,807 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:30:02,808 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:30:02,979 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:30:03,979 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:03,999 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:30:04,015 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:30:04,016 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:30:04,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:30:04,642 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:04,660 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:04,980 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:05,980 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:07,122 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:07,684 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:07,796 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:07,981 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:09,820 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:09,832 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:09,982 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:11,852 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:11,864 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:11,983 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:12,778 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:13,983 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:14,804 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:14,814 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:15,984 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:16,835 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:16,848 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:17,780 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:17,985 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:18,869 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:18,880 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:19,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:30:19,016 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:30:19,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:30:19,985 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:20,899 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:20,914 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:21,986 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:22,809 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:23,842 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:23,854 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:23,986 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:25,876 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:25,891 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:25,987 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:27,911 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:27,922 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:27,987 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:28,807 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:29,988 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:31,990 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:32,322 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:30:32,608 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:32,618 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:33,828 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:33,991 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:34,001 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:30:34,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:30:34,016 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:30:34,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:30:34,639 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:34,653 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:35,992 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:37,660 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:37,675 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:37,993 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:38,845 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:39,714 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:39,727 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:39,994 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:41,757 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:41,771 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:41,995 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:43,870 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:43,996 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:44,490 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:44,511 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:44,568 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:30:44,572 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:30:44,573 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:30:44,574 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:30:44,996 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:30:45,997 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:46,542 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:46,552 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:46,997 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:47,997 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:49,017 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:30:49,018 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:30:49,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:30:49,237 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:49,587 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:49,610 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:51,636 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:51,656 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:52,000 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:54,000 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:54,312 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:54,575 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:54,610 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:56,001 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:56,632 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:56,647 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:58,002 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:30:59,317 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:30:59,626 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:30:59,648 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:00,002 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:01,669 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:01,685 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:02,003 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:04,002 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:31:04,004 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:04,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:31:04,016 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:31:04,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:31:04,575 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:04,590 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:04,992 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:06,004 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:06,640 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:06,649 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:08,005 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:10,006 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:10,217 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:31:10,238 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:10,249 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:10,988 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:12,007 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:12,929 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:12,950 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:14,007 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:14,971 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:14,988 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:16,008 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:16,009 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:17,919 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:17,940 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:18,009 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:19,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:31:19,017 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:31:19,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:31:19,961 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:19,976 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:20,010 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:21,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:22,010 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:22,291 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:22,310 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:24,011 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:24,355 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:24,366 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:26,012 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:26,693 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:27,314 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:27,332 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:28,013 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:28,368 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:31:28,370 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:31:28,370 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:31:28,371 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:31:29,013 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:31:29,372 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:29,384 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:30,014 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:31,014 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:31,708 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:31,761 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:31,782 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:32,014 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:34,004 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:31:34,015 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:34,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:31:34,017 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:31:34,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:31:34,571 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:34,592 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:36,016 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:36,614 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:36,637 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:37,698 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:38,017 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:39,605 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:39,629 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:40,018 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:42,018 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:42,427 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:42,455 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:43,357 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:44,019 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:45,254 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:45,277 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:46,020 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:48,021 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:48,358 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:49,017 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:31:49,017 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:31:49,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:31:49,046 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:31:49,075 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:49,088 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:50,022 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:51,951 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:51,977 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:52,022 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:53,367 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:31:54,007 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:54,017 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:54,023 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:56,023 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:56,070 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:56,084 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:58,024 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:31:58,877 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:58,899 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:31:59,358 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:00,025 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:00,936 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:00,959 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:02,025 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:03,825 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:03,852 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:04,005 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:32:04,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:32:04,017 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:32:04,026 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:04,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:32:05,008 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:05,893 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:05,907 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:06,027 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:08,027 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:09,024 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:09,043 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:10,015 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:10,023 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:32:10,024 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:32:10,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:32:10,026 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:32:10,028 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:10,028 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:32:11,070 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:11,084 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:12,029 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:13,029 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:14,029 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:14,081 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:14,097 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:15,027 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:16,030 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:16,139 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:16,158 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:18,030 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:18,196 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:18,206 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:19,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:32:19,017 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:32:19,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:32:20,031 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:20,742 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:20,923 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:20,943 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:22,032 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:22,968 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:22,988 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:24,033 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:25,832 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:26,033 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:27,743 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:32:27,767 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:27,775 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:28,034 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:29,810 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:29,822 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:30,035 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:31,046 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:32,035 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:32,354 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:32,374 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:34,007 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:32:34,017 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:32:34,017 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:32:34,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:32:34,036 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:34,414 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:34,434 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:36,037 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:36,097 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:37,215 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:37,241 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:38,037 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:39,269 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:39,282 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:40,038 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:41,114 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:41,313 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:41,335 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:42,038 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:44,039 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:44,259 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:44,274 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:46,317 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:46,331 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:46,463 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:48,040 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:49,017 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:32:49,017 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:32:49,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:32:49,479 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:49,494 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:50,040 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:50,655 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:32:50,656 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:32:50,657 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:32:50,657 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:32:51,041 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:32:51,530 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:51,540 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:51,658 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:52,041 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:53,041 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:54,042 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:54,526 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:54,542 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:56,042 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:56,585 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:56,598 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:57,314 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:32:58,043 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:32:58,645 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:32:58,660 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:00,046 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:01,424 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:01,439 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:02,046 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:02,319 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:04,009 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:33:04,017 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:33:04,018 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:33:04,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:33:04,047 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:06,048 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:06,442 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:33:06,464 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:06,476 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:08,049 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:08,307 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:09,374 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:09,393 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:10,050 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:11,419 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:11,430 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:12,050 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:13,321 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:14,051 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:14,476 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:14,502 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:16,052 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:16,550 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:16,592 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:18,053 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:18,332 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:18,871 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:18,892 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:19,018 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:33:19,019 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:33:19,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:33:20,053 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:21,202 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:21,225 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:22,056 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:23,366 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:23,651 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:23,667 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:24,058 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:25,692 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:25,716 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:26,058 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:28,059 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:28,609 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:28,764 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:28,783 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:30,060 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:30,818 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:30,831 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:32,060 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:33,610 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:33:33,612 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:33:33,612 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:33:33,612 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:33,613 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:33:33,913 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:33,932 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:34,011 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:33:34,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:33:34,019 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:33:34,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:33:34,060 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:34,061 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:33:35,061 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:35,960 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:35,977 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:38,061 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:38,923 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:38,934 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:38,972 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:40,062 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:40,974 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:40,987 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:42,062 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:44,063 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:44,634 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:33:44,667 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:44,679 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:44,958 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:46,063 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:47,611 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:47,630 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:48,064 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:49,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:33:49,020 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:33:49,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:33:49,679 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:49,689 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:50,064 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:50,222 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:52,065 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:52,609 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:52,627 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:54,066 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:54,648 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:54,664 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:55,625 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:33:56,066 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:57,701 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:57,725 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:58,067 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:33:59,755 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:33:59,771 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:00,068 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:00,631 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:02,068 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:02,823 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:02,859 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:04,013 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:34:04,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:34:04,019 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:34:04,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:34:04,069 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:04,893 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:04,917 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:06,069 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:06,630 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:06,949 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:06,971 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:08,070 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:09,776 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:09,797 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:10,071 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:11,830 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:11,853 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:12,072 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:12,299 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:14,072 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:14,880 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:14,904 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:16,073 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:16,925 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:16,943 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:17,308 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:34:17,309 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:34:17,309 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:34:17,309 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:17,310 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:34:18,074 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:18,074 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:34:19,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:34:19,020 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:34:19,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:34:19,074 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:19,832 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:19,852 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:20,074 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:22,075 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:23,305 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:23,657 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:34:23,679 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:23,700 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:24,076 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:26,077 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:26,644 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:26,667 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:28,077 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:28,322 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:28,698 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:28,707 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:30,078 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:31,724 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:31,744 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:32,079 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:33,328 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:33,795 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:33,805 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:34,014 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:34:34,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:34:34,019 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:34:34,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:34:34,079 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:36,080 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:36,878 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:36,893 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:38,081 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:38,930 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:38,941 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:38,995 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:40,081 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:41,825 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:41,848 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:42,082 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:43,875 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:43,899 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:44,010 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:44,086 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:46,086 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:46,940 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:46,962 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:48,087 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:48,989 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:49,003 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:49,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:49,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:34:49,020 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:34:49,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:34:50,090 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:51,911 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:51,941 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:52,089 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:53,965 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:53,980 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:54,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:54,090 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:56,091 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:56,911 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:56,931 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:58,092 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:34:58,976 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:58,990 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:34:59,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:34:59,033 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:34:59,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:34:59,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:34:59,034 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:34:59,092 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:35:00,092 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:01,093 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:02,098 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:03,215 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:35:03,690 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:03,711 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:04,016 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:35:04,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:35:04,019 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:35:04,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:35:04,097 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:04,247 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:05,742 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:05,754 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:06,098 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:08,099 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:08,793 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:08,814 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:10,050 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:10,099 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:10,854 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:10,865 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:12,100 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:13,709 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:13,948 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:14,102 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:15,063 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:15,966 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:15,978 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:16,102 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:18,103 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:19,019 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:35:19,020 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:35:19,029 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:19,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:35:19,040 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:20,104 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:20,242 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:21,059 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:21,069 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:22,104 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:23,981 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:23,990 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:24,105 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:25,730 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:26,012 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:26,021 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:26,106 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:28,106 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:29,098 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:29,107 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:30,107 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:30,732 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:31,128 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:31,137 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:32,108 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:34,017 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:35:34,020 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:35:34,020 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:35:34,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:35:34,108 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:34,204 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:34,219 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:35,744 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:36,109 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:36,240 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:36,251 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:38,109 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:39,163 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:39,180 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:40,110 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:40,765 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:40,778 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:35:40,779 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:35:40,779 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:35:40,780 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:35:41,110 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:35:42,111 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:42,859 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:35:42,879 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:42,889 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:43,111 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:44,111 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:45,779 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:46,041 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:46,052 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:46,112 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:48,073 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:48,084 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:48,112 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:49,020 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:35:49,020 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:35:49,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:35:50,113 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:50,118 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:50,128 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:50,796 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:52,113 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:52,896 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:52,909 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:54,114 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:54,929 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:54,939 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:55,810 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:35:56,115 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:56,960 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:56,969 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:58,115 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:35:59,959 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:35:59,972 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:00,116 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:01,807 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:01,991 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:02,001 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:02,116 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:04,019 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:36:04,020 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:36:04,021 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:36:04,023 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:04,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:36:04,037 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:04,117 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:06,866 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:06,876 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:07,157 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:08,898 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:08,910 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:12,028 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:12,039 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:12,158 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:14,061 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:14,070 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:16,120 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:17,061 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:17,074 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:17,572 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:18,121 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:19,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:36:19,021 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:36:19,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:36:20,122 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:20,815 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:36:20,837 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:20,849 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:22,825 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:22,877 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:22,901 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:24,123 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:25,774 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:25,787 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:26,123 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:27,808 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:27,819 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:27,841 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:28,124 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:30,125 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:30,820 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:30,834 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:32,125 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:32,855 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:32,855 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:32,869 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:34,020 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:36:34,021 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:36:34,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:36:34,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:36:34,126 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:34,893 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:34,905 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:36,126 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:36,224 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:36:36,225 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:36:36,225 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:36:36,226 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:36:37,127 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:36:37,871 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:37,900 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:37,916 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:38,127 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:39,128 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:39,939 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:39,949 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:40,128 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:42,128 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:42,899 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:43,083 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:43,094 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:44,129 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:45,117 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:45,126 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:46,130 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:47,914 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:48,110 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:48,120 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:48,133 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:49,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:36:49,021 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:36:49,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:36:50,134 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:50,140 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:50,152 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:52,135 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:52,930 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:53,192 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:53,203 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:54,135 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:55,224 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:55,235 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:56,139 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:57,939 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:36:58,140 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:36:58,808 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:36:59,890 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:36:59,907 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:00,141 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:01,931 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:01,973 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:02,142 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:02,959 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:03,995 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:04,006 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:04,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:37:04,022 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:37:04,023 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:37:04,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:37:04,142 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:06,026 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:06,036 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:06,143 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:07,983 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:08,058 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:08,070 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:08,145 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:10,146 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:11,242 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:11,252 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:12,151 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:13,018 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:13,273 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:13,286 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:14,151 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:15,309 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:15,323 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:16,154 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:18,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:18,048 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:37:18,049 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:37:18,049 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:37:18,050 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:37:18,155 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:18,156 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:37:18,252 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:18,268 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:19,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:37:19,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:37:19,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:37:19,157 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:20,159 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:20,294 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:20,307 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:22,160 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:22,333 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:22,346 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:24,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:24,161 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:25,536 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:25,552 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:26,161 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:27,572 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:27,583 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:28,162 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:29,310 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:29,603 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:29,617 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:30,163 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:32,164 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:32,783 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:32,797 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:34,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:37:34,021 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:37:34,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:37:34,164 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:34,249 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:37:35,250 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:36,669 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:37:36,691 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:36,700 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:38,749 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:38,770 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:40,166 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:40,912 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:41,624 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:41,633 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:43,167 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:43,656 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:43,666 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:45,168 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:45,965 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:46,683 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:46,696 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:47,169 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:48,718 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:48,729 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:49,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:37:49,021 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:37:49,064 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:37:49,169 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:50,750 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:50,765 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:51,016 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:51,170 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:52,788 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:52,802 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:53,171 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:54,824 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:54,841 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:55,171 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:56,048 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:37:56,864 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:56,879 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:37:57,172 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:37:59,173 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:00,025 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:00,036 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:01,075 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:01,174 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:02,058 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:02,070 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:03,174 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:04,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:38:04,021 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:38:04,034 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:38:04,092 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:04,103 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:04,202 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:38:05,137 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:38:05,138 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:38:05,138 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:38:05,138 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:38:05,175 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:05,175 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:38:06,139 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:07,142 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:07,159 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:07,176 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:09,176 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:09,180 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:09,190 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:11,177 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:11,802 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:12,864 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:38:12,888 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:12,904 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:13,178 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:15,178 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:15,807 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:15,833 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:16,824 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:17,179 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:17,869 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:17,890 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:19,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:38:19,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:38:19,034 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:38:19,180 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:21,028 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:21,050 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:21,180 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:21,864 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:23,087 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:23,100 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:23,181 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:25,143 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:25,158 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:25,182 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:26,910 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:27,181 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:27,182 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:27,192 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:29,183 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:29,946 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:29,962 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:31,184 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:31,951 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:33,184 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:33,265 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:33,282 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:34,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:38:34,021 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:38:34,065 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:38:34,234 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:38:35,185 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:35,324 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:35,340 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:36,994 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:37,186 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:37,371 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:37,383 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:39,186 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:40,441 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:40,471 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:41,187 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:42,027 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:42,496 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:42,511 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:43,188 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:45,189 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:45,542 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:45,558 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:47,089 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:47,102 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:38:47,103 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:38:47,104 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:38:47,104 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:38:47,189 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:47,190 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:38:47,579 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:47,600 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:49,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:38:49,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:38:49,034 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:38:49,190 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:51,191 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:51,287 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:38:51,325 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:51,339 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:52,131 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:53,191 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:54,207 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:54,229 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:55,192 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:56,256 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:56,270 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:57,187 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:38:57,193 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:59,193 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:38:59,290 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:38:59,310 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:01,194 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:02,100 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:02,126 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:02,229 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:03,197 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:04,021 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:39:04,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:39:04,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:39:04,151 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:04,169 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:04,246 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:39:05,197 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:06,226 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:06,248 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:07,200 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:07,259 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:08,294 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:08,307 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:09,201 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:11,201 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:11,246 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:11,266 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:12,297 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:13,202 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:13,301 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:13,321 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:15,203 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:16,257 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:16,274 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:17,203 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:17,340 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:18,295 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:18,316 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:19,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:39:19,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:39:19,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:39:19,204 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:21,205 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:21,501 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:21,515 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:22,379 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:23,205 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:23,569 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:23,581 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:25,209 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:26,635 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:26,658 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:27,210 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:27,429 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:29,112 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 14:39:29,112 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 14:39:29,113 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 14:39:29,113 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 14:39:29,211 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:29,211 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 14:39:30,376 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:39:30,409 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:30,421 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:31,212 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:33,116 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:33,492 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:33,517 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:34,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:39:34,023 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:39:34,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:39:34,146 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:39:35,559 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:35,574 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:38,148 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:38,635 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:38,660 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:40,719 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:40,733 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:42,771 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:42,788 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:43,149 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:45,551 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:45,570 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:47,592 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:47,615 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:48,149 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:49,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:39:49,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:39:49,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:39:50,476 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:50,497 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:51,219 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:39:52,532 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:52,554 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:53,606 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:39:55,584 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:55,607 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:57,637 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:57,655 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:39:58,607 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:00,711 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:00,729 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:02,756 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:02,772 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:03,222 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:40:04,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:04,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:40:04,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:40:04,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:40:04,251 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:40:05,762 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:05,787 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:09,253 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:09,497 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:40:09,525 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:09,537 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:12,386 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:12,404 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:14,253 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:14,439 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:14,449 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:16,499 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:17,226 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:40:17,678 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:19,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:40:19,022 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:40:19,064 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:40:19,697 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:19,708 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:20,325 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:22,693 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:22,704 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:24,726 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:24,736 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:25,326 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:27,767 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:27,780 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:29,798 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:29,810 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:31,164 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:31,231 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:40:31,833 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:33,028 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:34,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:40:34,023 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:40:34,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:40:34,250 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:40:35,049 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:35,058 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:36,252 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:37,076 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:37,086 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:39,869 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:39,881 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:41,252 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:41,902 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:41,911 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:45,128 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:45,139 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:45,235 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:40:46,545 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:48,813 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:40:48,830 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:48,845 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:49,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:40:49,023 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:40:49,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:40:50,865 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:50,874 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:52,229 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:52,905 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:52,916 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:55,783 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:55,796 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:57,698 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:40:57,818 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:57,831 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:40:59,239 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:41:00,860 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:00,872 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:02,698 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:02,895 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:02,905 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:04,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:41:04,023 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:41:04,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:41:04,166 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:41:05,994 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:06,011 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:08,033 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:08,042 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:08,167 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:11,074 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:11,084 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:11,242 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:41:13,103 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:13,116 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:13,874 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:16,151 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:16,161 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:18,181 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:18,193 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:18,874 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:19,022 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:41:19,023 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:41:19,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:41:21,318 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:21,328 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:23,350 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:23,359 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:24,180 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:25,246 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:41:27,073 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:41:27,103 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:27,116 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:29,181 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:29,919 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:29,940 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:31,960 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:31,982 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:34,023 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:41:34,023 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:41:34,065 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:41:34,169 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:41:35,067 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:35,088 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:35,170 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:37,111 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:37,131 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:37,249 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:41:40,193 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:40,213 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:40,375 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:42,234 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:42,251 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:45,178 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:45,199 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:45,376 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:47,244 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:47,255 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:49,027 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:41:49,027 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:41:49,036 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:41:50,271 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:50,293 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:51,041 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:51,253 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:41:52,316 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:52,338 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:55,471 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:55,495 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:56,042 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:41:57,538 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:41:57,552 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:00,471 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:00,489 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:01,044 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:02,516 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:02,529 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:04,023 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:42:04,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:42:04,036 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:42:04,169 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:42:05,258 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:42:06,170 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:06,700 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:42:07,245 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:07,271 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:09,312 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:09,323 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:11,171 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:12,216 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:12,245 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:14,280 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:14,301 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:16,172 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:16,320 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:16,334 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:19,023 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:42:19,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:42:19,065 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:42:19,137 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:19,157 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:19,264 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:42:21,180 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:21,181 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:21,204 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:24,257 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:24,272 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:26,181 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:26,326 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:26,338 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:28,380 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:28,391 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:31,268 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:42:31,335 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:31,356 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:31,773 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:33,411 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:33,420 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:34,023 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:42:34,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:42:34,036 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:42:34,213 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:42:35,462 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:36,041 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:37,214 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:38,273 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:38,294 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:40,318 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:40,327 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:42,215 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:43,858 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:42:44,976 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:44,990 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:45,271 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:42:47,013 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:47,023 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:47,949 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:49,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:42:49,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:42:49,036 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:42:49,047 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:50,157 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:52,185 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:52,200 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:53,826 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:42:54,222 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:54,234 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:57,044 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:57,055 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:57,275 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:42:59,077 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:59,088 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:42:59,181 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:01,109 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:02,256 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:04,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:43:04,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:43:04,065 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:43:04,177 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:43:04,275 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:04,285 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:05,179 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:06,304 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:06,313 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:09,322 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:09,332 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:10,625 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:11,279 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:43:11,353 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:11,365 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:13,386 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:13,398 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:15,626 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:16,176 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:16,188 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:18,210 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:18,223 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:19,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:43:19,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:43:19,036 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:43:21,505 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:21,988 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:43:22,008 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:22,023 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:25,118 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:25,128 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:25,283 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:43:26,901 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:27,150 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:27,160 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:30,361 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:30,371 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:31,902 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:32,391 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:32,403 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:34,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:43:34,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:43:34,036 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:43:34,177 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:43:35,372 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:35,383 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:37,110 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:37,286 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:43:37,401 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:37,410 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:40,521 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:40,540 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:42,111 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:42,560 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:42,570 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:44,590 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:44,602 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:47,111 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:47,578 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:47,589 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:49,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:43:49,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:43:49,064 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:43:49,610 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:49,622 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:51,289 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:43:51,647 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:51,658 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:52,321 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:43:54,373 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:54,383 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:56,404 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:56,415 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:43:57,322 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:00,096 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:44:00,123 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:00,133 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:03,003 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:03,013 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:03,151 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:03,293 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:44:04,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:44:04,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:44:04,036 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:44:04,188 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:44:05,032 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:05,042 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:07,075 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:07,640 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:08,189 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:09,998 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:10,008 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:12,029 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:12,038 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:13,189 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:15,002 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:15,014 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:17,033 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:17,044 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:17,296 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:44:18,494 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:19,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:44:19,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:44:19,036 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:44:20,167 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:20,184 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:22,206 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:22,218 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:24,057 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:25,145 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:25,158 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:27,176 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:27,186 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:29,685 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:30,260 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:30,276 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:31,302 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:44:32,296 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:32,307 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:34,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:44:34,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:44:34,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:44:34,175 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:44:35,176 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:35,374 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:35,385 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:39,036 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:44:39,057 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:39,067 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:40,177 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:42,157 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:42,169 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:43,309 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:44:44,190 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:44,201 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:45,914 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:47,272 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:47,292 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:49,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:44:49,024 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:44:49,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:44:49,312 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:49,329 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:51,358 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:51,381 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:51,522 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:54,188 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:54,206 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:56,252 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:56,263 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:57,039 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:44:57,314 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:44:59,214 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:44:59,232 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:01,259 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:01,276 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:02,040 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:04,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:45:04,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:45:04,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:45:04,106 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:04,132 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:04,179 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:45:06,156 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:06,175 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:07,180 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:09,312 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:09,318 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:45:09,342 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:11,363 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:11,375 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:12,409 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:14,482 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:14,494 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:17,410 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:18,299 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:45:18,319 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:18,334 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:19,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:45:19,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:45:19,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:45:20,357 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:20,368 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:22,643 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:23,248 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:23,260 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:23,322 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:45:25,282 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:25,296 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:27,644 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:28,207 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:28,217 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:30,241 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:30,252 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:32,275 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:32,292 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:32,644 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:34,024 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:45:34,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:45:34,068 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:45:34,203 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:45:35,276 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:35,312 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:37,326 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:45:37,332 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:37,349 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:37,963 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:39,380 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:39,395 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:42,291 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:42,400 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:42,963 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:44,421 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:44,434 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:47,362 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:47,374 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:47,964 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:49,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:45:49,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:45:49,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:45:49,330 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:45:49,392 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:49,406 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:51,429 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:51,440 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:53,224 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:54,934 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:45:54,959 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:54,972 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:57,899 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:57,913 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:58,224 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:45:59,933 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:45:59,946 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:01,971 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:01,982 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:03,333 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:46:03,609 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:04,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:46:04,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:46:04,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:46:04,181 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:46:04,649 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:04,660 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:06,681 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:06,690 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:08,708 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:08,717 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:09,182 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:11,736 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:11,748 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:13,768 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:13,779 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:14,183 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:15,800 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:15,814 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:17,337 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:46:18,833 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:18,843 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:19,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:46:19,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:46:19,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:46:19,223 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:20,863 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:20,872 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:23,936 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:23,955 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:24,224 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:25,976 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:25,987 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:29,053 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:29,069 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:29,341 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:46:30,181 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:32,800 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:46:32,820 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:32,831 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:34,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:46:34,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:46:34,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:46:34,160 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:46:35,912 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:35,944 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:36,162 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:37,963 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:37,973 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:40,014 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:40,026 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:41,162 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:42,903 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:42,922 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:43,346 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:46:44,940 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:44,950 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:46,352 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:46,972 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:46,983 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:49,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:46:49,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:46:49,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:46:50,060 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:50,070 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:52,052 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:52,094 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:52,104 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:54,127 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:54,138 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:55,349 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:46:57,067 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:57,078 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:57,577 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:46:59,099 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:46:59,111 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:01,132 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:01,143 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:02,578 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:03,905 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:03,919 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:04,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:47:04,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:47:04,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:47:04,172 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:47:05,938 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:05,949 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:07,625 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:09,350 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:47:09,352 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:47:09,375 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:09,386 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:12,150 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:12,163 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:12,633 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:14,186 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:14,197 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:17,104 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:17,122 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:17,635 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:19,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:47:19,025 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:47:19,069 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:47:19,149 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:19,158 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:21,357 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:47:22,185 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:22,196 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:22,710 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:24,217 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:24,228 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:27,309 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:27,320 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:27,710 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:29,348 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:29,358 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:32,376 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:32,386 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:32,711 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:34,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:47:34,026 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:47:34,037 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:47:34,215 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:47:34,406 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:34,417 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:35,363 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:47:36,440 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:36,451 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:38,216 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:39,176 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:39,187 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:41,210 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:41,220 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:43,217 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:44,098 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:44,115 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:47,841 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:47:47,858 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:47,873 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:48,224 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:49,025 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:47:49,026 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:47:49,038 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:47:49,370 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:47:50,899 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:50,911 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:52,933 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:52,942 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:54,160 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:47:55,972 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:56,008 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:58,031 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:58,040 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:47:59,161 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:01,110 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:01,125 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:01,373 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:48:03,146 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:03,155 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:04,026 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:48:04,026 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:48:04,038 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:48:04,224 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:48:04,224 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:05,177 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:05,195 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:08,135 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:08,146 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:09,225 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:10,164 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:10,174 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:12,193 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:12,202 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:14,225 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:14,238 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:14,541 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:15,378 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:48:17,058 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:17,073 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:19,026 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:48:19,026 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:48:19,038 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:48:19,093 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:19,102 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:20,214 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:21,123 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:21,137 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:25,215 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:25,240 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:48:25,819 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:25,853 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:27,381 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:48:27,873 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:27,885 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:29,914 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:29,929 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:30,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:32,857 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:32,964 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:34,026 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:48:34,026 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:48:34,038 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:48:34,166 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:48:34,985 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:34,996 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:36,167 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:37,973 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:37,984 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:40,007 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:40,020 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:41,385 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:48:42,034 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:43,124 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:43,135 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:45,155 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:45,168 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:47,034 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:47,191 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:47,200 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:49,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:48:49,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:48:49,038 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:48:50,188 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:50,200 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:52,202 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:52,219 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:52,230 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:54,252 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:54,261 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:55,389 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:48:57,212 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:57,218 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:48:57,226 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:59,249 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:48:59,259 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:02,218 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:02,880 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:49:02,912 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:02,931 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:04,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:49:04,029 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:49:04,072 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:49:04,255 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:49:05,835 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:05,846 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:07,393 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:49:07,464 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:07,867 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:07,876 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:09,909 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:09,922 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:12,465 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:12,917 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:12,934 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:14,952 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:14,961 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:16,993 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:17,004 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:17,465 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:19,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:49:19,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:49:19,038 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:49:20,019 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:20,030 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:21,397 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:49:22,049 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:22,064 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:22,571 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:25,199 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:25,223 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:27,242 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:27,261 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:27,572 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:30,392 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:30,418 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:32,439 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:32,466 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:32,573 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:34,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:49:34,029 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:49:34,038 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:49:34,220 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:49:34,492 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:35,294 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:35,400 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:49:37,398 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:37,409 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:38,221 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:41,081 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:49:41,104 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:41,114 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:43,222 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:44,088 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:44,113 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:46,138 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:46,162 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:47,404 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:49:48,196 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:48,219 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:49,027 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:49,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:49:49,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:49:49,072 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:49:51,099 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:51,113 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:53,164 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:53,175 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:54,184 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:55,201 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:55,214 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:57,235 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:57,250 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:59,184 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:49:59,275 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:49:59,289 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:01,314 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:01,326 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:01,408 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:50:03,360 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:03,382 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:04,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:50:04,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:50:04,039 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:50:04,195 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:50:04,196 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:05,401 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:05,411 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:08,279 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:08,297 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:09,197 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:10,335 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:10,363 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:12,385 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:12,396 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:14,509 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:15,411 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:50:16,177 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:50:16,209 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:16,225 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:19,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:50:19,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:50:19,039 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:50:19,418 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:19,438 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:20,176 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:21,465 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:21,476 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:23,500 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:23,522 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:25,177 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:26,334 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:26,355 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:28,376 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:28,400 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:28,415 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:50:30,720 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:31,386 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:31,414 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:33,465 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:33,475 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:34,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:50:34,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:50:34,073 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:50:34,251 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:50:36,252 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:36,463 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:36,486 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:38,515 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:38,528 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:41,617 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:41,639 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:41,917 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:42,419 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:50:43,667 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:43,689 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:46,539 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:46,556 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:46,919 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:48,588 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:48,603 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:49,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:50:49,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:50:49,040 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:50:51,642 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:51,661 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:52,194 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:55,368 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:50:55,386 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:55,400 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:56,425 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:50:57,358 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:50:58,444 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:50:58,459 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:00,495 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:00,507 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:02,358 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:03,628 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:03,652 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:04,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:51:04,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:51:04,039 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:51:04,228 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:51:05,697 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:05,711 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:07,434 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:07,740 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:07,763 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:08,428 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:51:10,911 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:10,922 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:12,434 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:12,959 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:12,969 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:16,157 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:16,177 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:17,435 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:18,197 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:18,215 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:19,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:51:19,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:51:19,073 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:51:20,238 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:20,251 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:22,433 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:51:22,985 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:23,004 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:23,270 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:25,033 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:25,048 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:28,181 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:28,202 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:28,271 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:30,222 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:30,239 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:33,936 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:51:33,960 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:33,986 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:34,028 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:34,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:51:34,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:51:34,039 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:51:34,198 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:51:34,437 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:51:36,961 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:36,984 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:39,017 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:39,045 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:39,200 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:41,085 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:41,111 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:43,937 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:43,951 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:44,201 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:45,998 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:46,007 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:48,441 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:51:49,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:51:49,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:51:49,039 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:51:49,063 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:49,076 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:49,223 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:51,113 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:51,125 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:53,169 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:53,183 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:54,224 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:51:56,179 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:56,197 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:58,218 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:58,239 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:51:59,225 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:00,271 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:00,288 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:02,445 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:52:03,188 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:03,215 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:04,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:52:04,030 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:52:04,073 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:52:04,256 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:52:04,257 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:05,239 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:05,256 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:08,299 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:08,320 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:09,258 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:12,008 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:52:12,029 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:12,041 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:14,259 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:15,163 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:15,179 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:16,449 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:52:17,199 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:17,210 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:19,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:52:19,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:52:19,040 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:52:20,344 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:20,354 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:21,000 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:22,376 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:22,387 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:24,410 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:24,420 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:26,001 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:27,440 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:27,449 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:29,469 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:29,478 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:30,454 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:52:31,425 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:32,619 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:32,643 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:34,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:52:34,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:52:34,040 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:52:34,218 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:52:34,666 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:34,678 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:36,699 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:36,719 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:37,219 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:39,719 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:39,738 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:41,768 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:41,780 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:42,703 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:43,806 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:43,818 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:44,458 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:52:46,890 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:46,900 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:47,704 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:49,030 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:52:49,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:52:49,073 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:52:50,603 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:52:50,629 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:50,639 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:52,659 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:52,677 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:53,215 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:52:55,585 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:55,603 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:56,462 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:52:57,647 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:57,656 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:52:59,013 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:00,776 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:00,797 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:02,818 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:02,833 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:04,014 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:04,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:53:04,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:53:04,040 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:53:04,257 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:53:04,864 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:04,889 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:07,777 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:07,795 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:09,257 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:09,820 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:09,837 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:10,466 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:53:11,860 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:11,885 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:14,258 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:14,882 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:14,904 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:16,926 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:16,944 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:18,975 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:18,993 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:19,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:53:19,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:53:19,040 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:53:20,164 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:21,789 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:21,800 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:23,823 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:23,840 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:24,470 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:53:25,415 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:27,458 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:53:27,485 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:27,501 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:30,416 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:30,438 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:30,450 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:32,472 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:53:32,473 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:32,485 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:34,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:53:34,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:53:34,073 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:53:34,206 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:53:35,403 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:35,420 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:36,208 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:37,439 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:37,453 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:38,474 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:53:40,497 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:40,508 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:41,914 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:42,531 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:42,542 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:45,546 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:45,556 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:46,477 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:53:47,249 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:47,575 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:47,586 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:49,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:53:49,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:53:49,040 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:53:50,537 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:50,551 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:52,608 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:52,621 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:53,198 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:53:55,630 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:55,646 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:57,686 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:57,695 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:53:58,199 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:00,481 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:54:00,662 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:00,673 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:02,692 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:02,707 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:03,526 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:04,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:54:04,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:54:04,042 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:54:04,217 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:54:06,368 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:54:06,388 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:06,399 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:09,183 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:09,205 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:09,219 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:11,228 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:11,247 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:14,336 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:14,348 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:14,487 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:54:14,570 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:16,385 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:16,394 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:19,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:54:19,031 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:54:19,073 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:54:19,241 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:19,266 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:20,951 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:21,300 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:21,328 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:24,426 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:24,449 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:26,468 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:26,486 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:26,491 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:54:26,893 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:29,850 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:29,870 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:31,894 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:31,904 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:31,922 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:33,966 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:33,981 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:34,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:54:34,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:54:34,041 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:54:34,255 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:54:36,948 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:36,967 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:37,260 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:38,988 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:38,999 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:40,496 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:54:41,026 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:41,039 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:42,385 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:44,368 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:54:44,387 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:44,398 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:47,335 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:47,346 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:47,386 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:49,031 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:54:49,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:54:49,041 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:54:49,364 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:49,376 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:51,405 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:51,416 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:52,696 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:54,296 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:54,309 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:54,501 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:54:56,328 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:56,338 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:57,697 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:54:58,357 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:54:58,367 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:00,389 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:00,399 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:02,698 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:03,363 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:03,373 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:04,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:55:04,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:55:04,072 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:55:04,264 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:55:05,396 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:05,409 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:06,504 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:55:07,436 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:07,453 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:08,112 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:10,269 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:10,295 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:12,315 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:12,335 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:13,113 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:14,360 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:15,492 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:17,515 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:17,527 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:18,114 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:19,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:55:19,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:55:19,041 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:55:20,508 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:55:21,259 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:55:21,280 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:21,292 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:23,312 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:23,328 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:23,819 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:26,343 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:26,360 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:28,399 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:28,413 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:28,820 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:30,456 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:31,264 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:33,283 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:33,298 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:34,029 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:34,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:55:34,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:55:34,041 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:55:34,174 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:55:34,512 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:55:35,324 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:35,334 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:38,394 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:38,404 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:39,176 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:40,429 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:40,441 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:42,464 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:42,477 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:44,176 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:45,463 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:45,475 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:46,516 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:55:47,494 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:47,503 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:49,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:55:49,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:55:49,072 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:55:49,184 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:49,529 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:49,544 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:52,473 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:52,483 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:54,185 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:55:54,502 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:54,513 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:58,146 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:55:58,165 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:58,174 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:55:59,303 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:00,520 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:56:01,217 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:01,227 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:03,245 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:03,254 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:04,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:56:04,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:56:04,041 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:56:04,197 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:56:05,198 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:05,505 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:05,519 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:08,682 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:08,691 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:10,198 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:10,710 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:10,726 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:12,748 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:12,761 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:14,524 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:56:15,698 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:15,728 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:15,739 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:17,758 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:17,773 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:19,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:56:19,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:56:19,041 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:56:20,854 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:20,872 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:20,882 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:22,902 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:22,920 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:24,949 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:24,972 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:26,527 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:56:26,788 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:26,996 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:27,012 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:29,941 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:29,951 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:31,789 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:31,972 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:31,982 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:34,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:56:34,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:56:34,072 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:56:34,213 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:56:35,711 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:56:35,730 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:35,743 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:37,215 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:38,731 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:38,741 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:40,531 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:56:40,758 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:40,770 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:42,794 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:42,805 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:43,151 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:45,769 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:45,789 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:47,813 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:47,822 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:48,151 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:49,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:56:49,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:56:49,042 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:56:50,790 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:50,800 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:52,534 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:56:52,829 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:52,838 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:53,321 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:56:55,911 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:55,921 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:57,941 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:57,950 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:56:58,322 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:00,917 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:00,928 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:02,951 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:02,964 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:03,323 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:04,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:57:04,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:57:04,042 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:57:04,175 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:57:04,989 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:05,011 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:06,538 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:57:07,847 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:07,863 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:08,487 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:09,884 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:09,895 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:13,489 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:13,569 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:57:13,590 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:13,609 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:16,436 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:16,447 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:18,469 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:18,481 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:18,725 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:19,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:57:19,032 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:57:19,072 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:57:20,543 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:57:21,521 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:21,531 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:23,664 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:23,674 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:24,249 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:25,694 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:25,707 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:27,732 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:27,742 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:29,250 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:30,673 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:30,688 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:32,549 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:57:32,709 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:32,719 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:34,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:57:34,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:57:34,042 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:57:34,248 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:57:35,250 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:35,643 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:35,653 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:37,672 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:37,682 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:40,250 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:40,484 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:40,505 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:42,525 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:42,534 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:44,555 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:44,566 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:46,217 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:46,553 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:57:47,545 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:47,556 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:49,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:57:49,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:57:49,042 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:57:51,283 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:57:51,307 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:51,317 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:51,558 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:53,369 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:53,379 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:55,421 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:55,434 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:56,558 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:57:58,455 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:58,484 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:57:58,558 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:58:00,512 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:00,550 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:02,523 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:03,584 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:03,623 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:04,032 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:58:04,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:58:04,073 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:58:04,184 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:58:05,654 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:05,687 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:08,186 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:08,717 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:08,745 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:10,768 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:10,792 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:12,562 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:58:12,811 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:12,835 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:13,728 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:15,985 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:16,007 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:18,050 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:18,072 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:18,728 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:19,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:58:19,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:58:19,042 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:58:21,188 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:21,201 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:23,231 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:23,252 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:23,896 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:25,284 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:25,294 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:26,566 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:58:28,938 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:29,960 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:58:30,000 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:30,016 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:32,061 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:32,074 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:33,939 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:34,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:58:34,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:58:34,042 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:58:34,125 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:34,136 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:34,169 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:58:37,071 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:37,086 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:38,570 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:58:39,034 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:39,124 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:39,136 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:41,163 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:41,173 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:44,035 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:44,224 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:44,245 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:46,270 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:46,289 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:48,343 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:49,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:58:49,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:58:49,077 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:58:49,189 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:49,626 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:51,645 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:51,655 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:52,574 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:58:53,677 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:53,691 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:54,317 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:58:56,581 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:56,591 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:58,611 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:58,624 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:58:59,318 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:01,731 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:01,743 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:03,763 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:03,774 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:04,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:59:04,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:59:04,043 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:59:04,194 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:59:04,577 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:59:05,196 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:07,484 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:59:07,504 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:07,517 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:10,197 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:10,561 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:10,571 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:12,590 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:12,602 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:14,627 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:14,642 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:15,197 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:17,658 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:17,669 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:18,581 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:59:19,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:59:19,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:59:19,043 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:59:19,689 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:19,703 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:20,706 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:21,728 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:21,743 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:24,434 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:24,449 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:25,706 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:26,497 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:26,519 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:28,547 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:28,559 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:31,003 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:31,585 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:31,606 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:32,586 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:59:33,636 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:33,648 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:34,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:59:34,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:59:34,076 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:59:34,206 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 14:59:35,688 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:35,701 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:36,208 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:38,621 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:38,642 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:40,661 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:40,671 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:41,208 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:44,363 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 14:59:44,383 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:44,397 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:44,589 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 14:59:46,256 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:46,416 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:46,425 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:49,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 14:59:49,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 14:59:49,043 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 14:59:49,375 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:49,396 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:51,416 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:51,434 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:52,138 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:53,461 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:53,479 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:56,457 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:56,481 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:57,449 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 14:59:58,528 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:58,540 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 14:59:58,593 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:00:01,672 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:01,695 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:02,450 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:03,715 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:03,728 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:04,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:00:04,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:00:04,043 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:00:04,198 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 15:00:05,748 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:05,767 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:08,199 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:08,827 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:08,848 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:10,597 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:00:10,870 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:10,883 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:12,915 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:12,937 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:13,578 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:15,886 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:15,906 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:17,949 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:17,967 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:18,579 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:19,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:00:19,034 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:00:19,077 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:00:21,412 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 15:00:21,438 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:21,451 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:23,632 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:24,366 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:24,377 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:24,601 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:00:26,395 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:26,405 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:28,437 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:28,634 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:29,517 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:31,536 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:31,547 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:33,635 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:34,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:00:34,033 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:00:34,043 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:00:34,181 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 15:00:34,658 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:34,679 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:36,706 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:36,719 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:38,605 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:00:38,909 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:39,733 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:39,755 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:41,798 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:41,807 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:43,910 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:44,776 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:44,803 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:46,829 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:46,846 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:48,911 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:49,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:00:49,034 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:00:49,043 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:00:49,948 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:49,962 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:52,006 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:52,016 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:52,613 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:00:54,074 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:00:55,075 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:55,106 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:57,127 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:57,140 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:00:59,074 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:00,855 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 15:01:00,898 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:00,916 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:03,806 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:03,831 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:04,033 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:01:04,034 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:01:04,076 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:01:04,192 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 15:01:04,193 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:04,624 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:05,854 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:05,869 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:08,832 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:08,852 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:09,193 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:10,892 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:10,902 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:11,781 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 15:01:11,782 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 15:01:11,783 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 15:01:11,783 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 15:01:11,783 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 15:01:11,784 DEBUG SenderThread:1483490 [sender.py:send():382] send: metric 2024-03-27 15:01:11,784 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 15:01:11,784 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 15:01:11,784 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 15:01:12,627 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:12,627 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 15:01:13,893 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:13,913 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:14,628 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:14,785 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:15,952 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:15,962 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:19,095 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:19,114 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:19,786 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:19,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:01:19,797 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:01:19,799 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:01:20,630 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:21,135 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:21,148 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:22,631 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:23,169 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:23,191 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:24,995 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:25,214 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:25,228 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:28,251 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:28,682 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:30,411 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:30,633 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:30,633 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/config.yaml 2024-03-27 15:01:31,806 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:31,832 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:32,634 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:33,856 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:33,874 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:34,101 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 15:01:34,635 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:34,796 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:01:34,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:01:34,797 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:01:35,895 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:35,912 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:35,939 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:36,635 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:38,636 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:39,606 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 15:01:39,634 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:39,648 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:40,636 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:41,183 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:42,416 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:42,428 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:42,637 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:44,476 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:44,494 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:44,638 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:46,234 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:46,532 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:46,558 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:46,638 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:48,639 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:49,557 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:49,596 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:49,796 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:01:49,797 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:01:49,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:01:50,640 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:51,643 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:51,670 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:51,930 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:52,640 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:53,739 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:53,764 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:54,641 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:56,642 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:56,893 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:56,929 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:57,000 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:01:58,642 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:01:58,976 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:01:58,989 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:00,643 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:01,052 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:01,067 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:02,058 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:02,644 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:04,103 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 15:02:04,163 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:04,175 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:04,645 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:04,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:02:04,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:02:04,798 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:02:06,234 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:06,256 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:06,646 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:07,097 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:08,277 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:08,301 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:08,646 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:10,647 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:11,073 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:11,094 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:11,168 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 15:02:11,169 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 15:02:11,169 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 15:02:11,170 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 15:02:11,647 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 15:02:12,171 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:12,648 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:13,113 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:13,132 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:14,648 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:16,649 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:16,869 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 15:02:16,896 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:16,906 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:17,209 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:18,650 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:18,936 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:18,946 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:19,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:02:19,797 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:02:19,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:02:20,651 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:20,980 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:21,610 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:22,274 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:22,652 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:23,967 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:23,978 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:24,652 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:25,997 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:26,010 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:26,653 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:27,356 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:28,035 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:28,047 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:28,654 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:30,655 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:31,076 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:31,088 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:32,412 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:32,655 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:33,110 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:33,120 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:34,104 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 15:02:34,656 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:34,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:02:34,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:02:34,797 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:02:35,142 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:35,153 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:36,657 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:37,470 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:37,971 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:37,988 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:38,658 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:40,019 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:40,030 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:40,659 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:42,050 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:42,066 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:42,512 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:42,659 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:44,660 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:45,086 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:45,098 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:46,660 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:47,126 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:47,145 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:47,569 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:48,661 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:49,167 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:49,177 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:49,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:02:49,798 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:02:49,798 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:02:50,662 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:52,650 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:52,662 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:53,062 ERROR gpu :1483490 [interfaces.py:aggregate():161] Failed to serialize metric: division by zero 2024-03-27 15:02:53,094 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:53,104 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:53,351 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: partial_history 2024-03-27 15:02:53,352 DEBUG SenderThread:1483490 [sender.py:send():382] send: history 2024-03-27 15:02:53,353 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: summary_record 2024-03-27 15:02:53,353 INFO SenderThread:1483490 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end 2024-03-27 15:02:53,663 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/wandb-summary.json 2024-03-27 15:02:54,663 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:56,057 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:56,068 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:56,664 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:02:57,720 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:02:58,093 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:58,103 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:02:58,664 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:03:00,122 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:00,136 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:00,665 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:03:02,158 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:02,170 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:02,666 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:03:02,772 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:03:04,106 DEBUG SenderThread:1483490 [sender.py:send():382] send: stats 2024-03-27 15:03:04,666 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:03:04,797 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: stop_status 2024-03-27 15:03:04,797 DEBUG SenderThread:1483490 [sender.py:send_request():409] send_request: stop_status 2024-03-27 15:03:04,799 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: internal_messages 2024-03-27 15:03:05,107 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:05,145 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:06,667 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:03:07,187 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:07,218 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:07,823 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report 2024-03-27 15:03:08,668 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:03:09,275 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:09,296 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:10,668 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:03:12,419 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:12,440 ERROR gpu :1483490 [interfaces.py:monitor():144] Failed to sample metric: Not Supported 2024-03-27 15:03:12,669 INFO Thread-13 :1483490 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/sanchit/distil-large-v3-hi-ft-frozen-encoder/wandb/run-20240327_141033-golaq7b9/files/output.log 2024-03-27 15:03:12,889 DEBUG HandlerThread:1483490 [handler.py:handle_request():146] handle_request: status_report