| 2023-11-18 12:32:18,522 INFO StreamThr :39554 [internal.py:wandb_internal():86] W&B internal server running at pid: 39554, started at: 2023-11-18 12:32:18.520673 | |
| 2023-11-18 12:32:18,524 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status | |
| 2023-11-18 12:32:18,527 INFO WriterThread:39554 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_123218-m1nos8ra/run-m1nos8ra.wandb | |
| 2023-11-18 12:32:18,529 DEBUG SenderThread:39554 [sender.py:send():380] send: header | |
| 2023-11-18 12:32:18,529 DEBUG SenderThread:39554 [sender.py:send():380] send: run | |
| 2023-11-18 12:32:19,029 INFO SenderThread:39554 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files | |
| 2023-11-18 12:32:19,036 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: check_version | |
| 2023-11-18 12:32:19,038 INFO SenderThread:39554 [sender.py:_start_run_threads():1124] run started: m1nos8ra with start time 1700303538.52072 | |
| 2023-11-18 12:32:19,039 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: check_version | |
| 2023-11-18 12:32:19,108 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: run_start | |
| 2023-11-18 12:32:19,124 DEBUG HandlerThread:39554 [system_info.py:__init__():32] System info init | |
| 2023-11-18 12:32:19,124 DEBUG HandlerThread:39554 [system_info.py:__init__():47] System info init done | |
| 2023-11-18 12:32:19,124 INFO HandlerThread:39554 [system_monitor.py:start():194] Starting system monitor | |
| 2023-11-18 12:32:19,125 INFO SystemMonitor:39554 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2023-11-18 12:32:19,125 INFO HandlerThread:39554 [system_monitor.py:probe():214] Collecting system info | |
| 2023-11-18 12:32:19,126 INFO SystemMonitor:39554 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-11-18 12:32:19,128 INFO SystemMonitor:39554 [interfaces.py:start():190] Started disk monitoring | |
| 2023-11-18 12:32:19,128 INFO SystemMonitor:39554 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-11-18 12:32:19,129 INFO SystemMonitor:39554 [interfaces.py:start():190] Started memory monitoring | |
| 2023-11-18 12:32:19,131 INFO SystemMonitor:39554 [interfaces.py:start():190] Started network monitoring | |
| 2023-11-18 12:32:19,138 DEBUG HandlerThread:39554 [system_info.py:probe():196] Probing system | |
| 2023-11-18 12:32:19,229 DEBUG HandlerThread:39554 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2023-11-18 12:32:19,229 DEBUG HandlerThread:39554 [system_info.py:probe():244] Probing system done | |
| 2023-11-18 12:32:19,230 DEBUG HandlerThread:39554 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T10:32:19.139216', 'startedAt': '2023-11-18T10:32:18.452354', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=1', '--per_device_eval_batch_size=1', '--gradient_accumulation_steps=2', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.41075134277344}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} | |
| 2023-11-18 12:32:19,230 INFO HandlerThread:39554 [system_monitor.py:probe():224] Finished collecting system info | |
| 2023-11-18 12:32:19,230 INFO HandlerThread:39554 [system_monitor.py:probe():227] Publishing system info | |
| 2023-11-18 12:32:19,230 DEBUG HandlerThread:39554 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |
| 2023-11-18 12:32:19,232 DEBUG HandlerThread:39554 [system_info.py:_save_pip():68] Saving pip packages done | |
| 2023-11-18 12:32:19,232 DEBUG HandlerThread:39554 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |
| 2023-11-18 12:32:20,033 INFO Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/requirements.txt | |
| 2023-11-18 12:32:20,034 INFO Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/conda-environment.yaml | |
| 2023-11-18 12:32:22,038 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/conda-environment.yaml | |
| 2023-11-18 12:32:22,039 DEBUG HandlerThread:39554 [system_info.py:_save_conda():87] Saving conda packages done | |
| 2023-11-18 12:32:22,041 INFO HandlerThread:39554 [system_monitor.py:probe():229] Finished publishing system info | |
| 2023-11-18 12:32:22,045 DEBUG SenderThread:39554 [sender.py:send():380] send: files | |
| 2023-11-18 12:32:22,045 INFO SenderThread:39554 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now | |
| 2023-11-18 12:32:22,052 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:32:22,053 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:32:22,055 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:32:22,221 DEBUG SenderThread:39554 [sender.py:send():380] send: telemetry | |
| 2023-11-18 12:32:22,221 DEBUG SenderThread:39554 [sender.py:send():380] send: config | |
| 2023-11-18 12:32:22,222 DEBUG SenderThread:39554 [sender.py:send():380] send: metric | |
| 2023-11-18 12:32:22,222 DEBUG SenderThread:39554 [sender.py:send():380] send: telemetry | |
| 2023-11-18 12:32:22,222 DEBUG SenderThread:39554 [sender.py:send():380] send: metric | |
| 2023-11-18 12:32:22,223 WARNING SenderThread:39554 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) | |
| 2023-11-18 12:32:22,223 DEBUG SenderThread:39554 [sender.py:send():380] send: telemetry | |
| 2023-11-18 12:32:22,658 INFO wandb-upload_0:39554 [upload_job.py:push():131] Uploaded file /tmp/tmpnmps1g75wandb/3bw550az-wandb-metadata.json | |
| 2023-11-18 12:32:23,042 INFO Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/wandb-metadata.json | |
| 2023-11-18 12:32:23,043 INFO Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:32:23,636 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:32:25,050 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:32:27,151 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:32:28,799 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:32:33,800 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:32:37,052 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:32:37,053 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:32:37,054 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:32:39,241 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:32:44,242 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:32:49,245 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:32:50,246 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/config.yaml | |
| 2023-11-18 12:32:52,053 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:32:52,053 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:32:52,105 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:32:55,307 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:00,307 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:05,308 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:07,053 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:33:07,053 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:33:07,095 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:33:11,243 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:16,244 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:19,131 DEBUG SystemMonitor:39554 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2023-11-18 12:33:19,132 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:33:22,053 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:33:22,054 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:33:22,095 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:33:22,249 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:27,250 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:32,250 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:37,053 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:33:37,054 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:33:37,095 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:33:38,214 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:43,215 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:48,216 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:49,133 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:33:52,053 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:33:52,054 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:33:52,095 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:33:53,313 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:33:58,314 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:03,315 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:07,053 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:34:07,054 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:34:07,095 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:34:09,288 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:14,290 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:19,133 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:34:20,134 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:22,054 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:34:22,054 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:34:22,057 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:34:22,621 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:34:24,629 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:34:25,332 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:30,333 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:35,334 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:37,054 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:34:37,054 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:34:37,095 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:34:41,269 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:46,270 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:49,134 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:34:52,056 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:34:52,056 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:34:52,057 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:34:52,057 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:34:57,312 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:02,313 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:07,055 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:35:08,167 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:08,167 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:35:08,168 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:35:12,462 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:18,404 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:18,775 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:35:20,025 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:35:22,264 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:35:22,264 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:35:23,178 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:35:23,560 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:28,560 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:33,561 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:37,263 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:35:37,264 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:35:38,178 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:35:39,538 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:44,540 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:49,173 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:35:50,005 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:35:50,018 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:35:52,265 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:35:52,266 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:35:53,179 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:35:55,215 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:35:55,525 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:01,076 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:01,254 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:05,281 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:06,419 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:08,013 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:36:08,015 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:36:08,179 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:36:11,325 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:12,003 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:17,373 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:17,577 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:20,018 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:36:21,401 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:23,014 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:36:23,017 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:36:23,324 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:24,575 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:36:27,439 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:28,604 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:33,476 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:34,182 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:38,013 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:36:38,014 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:36:39,232 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:39,517 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:39,574 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:36:41,526 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:43,537 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:44,906 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:48,573 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:50,019 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:36:50,020 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:36:53,014 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:36:53,015 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:36:54,596 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:36:54,609 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:36:55,856 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:00,657 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:01,795 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:06,703 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:07,726 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:08,013 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:37:08,014 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:37:10,322 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:37:12,745 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:13,317 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:18,747 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:18,788 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:20,022 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:37:22,815 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:23,013 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:37:23,014 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:37:24,220 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:26,592 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:37:28,855 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:29,538 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:34,893 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:35,252 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:38,014 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:37:38,016 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:37:40,723 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:40,937 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:41,594 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:37:46,255 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:46,990 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:49,763 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: partial_history | |
| 2023-11-18 12:37:49,765 DEBUG SenderThread:39554 [sender.py:send():380] send: metric | |
| 2023-11-18 12:37:49,766 DEBUG SenderThread:39554 [sender.py:send():380] send: metric | |
| 2023-11-18 12:37:49,766 DEBUG SenderThread:39554 [sender.py:send():380] send: metric | |
| 2023-11-18 12:37:49,766 DEBUG SenderThread:39554 [sender.py:send():380] send: history | |
| 2023-11-18 12:37:49,766 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: summary_record | |
| 2023-11-18 12:37:49,781 INFO SenderThread:39554 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end | |
| 2023-11-18 12:37:50,016 INFO Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/wandb-summary.json | |
| 2023-11-18 12:37:50,021 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:37:51,020 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:52,022 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:37:52,026 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:54,274 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:37:54,276 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:37:56,592 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:37:57,064 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:37:57,234 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:02,884 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:03,109 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:08,415 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:09,162 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:09,275 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:38:09,277 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:38:11,593 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:38:13,195 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:13,975 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:15,212 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/config.yaml | |
| 2023-11-18 12:38:19,243 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:19,528 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:20,032 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:38:24,274 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:38:24,274 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:38:25,299 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:25,424 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:27,645 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:38:30,645 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:31,346 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:35,373 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:36,109 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:39,275 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:38:39,276 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:38:41,415 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:41,752 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:42,645 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:38:47,464 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:47,717 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:50,023 DEBUG SenderThread:39554 [sender.py:send():380] send: stats | |
| 2023-11-18 12:38:53,105 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:53,538 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:38:55,752 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:38:55,755 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 12:38:57,645 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 12:38:58,775 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:38:59,601 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:39:04,384 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:39:05,645 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:39:09,673 INFO Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log | |
| 2023-11-18 12:39:10,194 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 12:39:10,751 DEBUG HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 12:39:10,752 DEBUG SenderThread:39554 [sender.py:send_request():407] send_request: stop_status | |