Hugging Face's logo

Finnish-NLP
/
whisper-large-finnish-v3

Automatic Speech Recognition
Transformers
TensorBoard
Safetensors
Finnish
whisper
whisper-event
finnish
speech-recognition
Eval Results
Inference Endpoints
Model card Files Metrics Community
1
whisper-large-finnish-v3
File size: 36,624 Bytes 
b0ae254
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
 
2023-11-18 12:32:18,522 INFO    StreamThr :39554 [internal.py:wandb_internal():86] W&B internal server running at pid: 39554, started at: 2023-11-18 12:32:18.520673
2023-11-18 12:32:18,524 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status
2023-11-18 12:32:18,527 INFO    WriterThread:39554 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_123218-m1nos8ra/run-m1nos8ra.wandb
2023-11-18 12:32:18,529 DEBUG   SenderThread:39554 [sender.py:send():380] send: header
2023-11-18 12:32:18,529 DEBUG   SenderThread:39554 [sender.py:send():380] send: run
2023-11-18 12:32:19,029 INFO    SenderThread:39554 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files
2023-11-18 12:32:19,036 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: check_version
2023-11-18 12:32:19,038 INFO    SenderThread:39554 [sender.py:_start_run_threads():1124] run started: m1nos8ra with start time 1700303538.52072
2023-11-18 12:32:19,039 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: check_version
2023-11-18 12:32:19,108 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: run_start
2023-11-18 12:32:19,124 DEBUG   HandlerThread:39554 [system_info.py:__init__():32] System info init
2023-11-18 12:32:19,124 DEBUG   HandlerThread:39554 [system_info.py:__init__():47] System info init done
2023-11-18 12:32:19,124 INFO    HandlerThread:39554 [system_monitor.py:start():194] Starting system monitor
2023-11-18 12:32:19,125 INFO    SystemMonitor:39554 [system_monitor.py:_start():158] Starting system asset monitoring threads
2023-11-18 12:32:19,125 INFO    HandlerThread:39554 [system_monitor.py:probe():214] Collecting system info
2023-11-18 12:32:19,126 INFO    SystemMonitor:39554 [interfaces.py:start():190] Started cpu monitoring
2023-11-18 12:32:19,128 INFO    SystemMonitor:39554 [interfaces.py:start():190] Started disk monitoring
2023-11-18 12:32:19,128 INFO    SystemMonitor:39554 [interfaces.py:start():190] Started gpu monitoring
2023-11-18 12:32:19,129 INFO    SystemMonitor:39554 [interfaces.py:start():190] Started memory monitoring
2023-11-18 12:32:19,131 INFO    SystemMonitor:39554 [interfaces.py:start():190] Started network monitoring
2023-11-18 12:32:19,138 DEBUG   HandlerThread:39554 [system_info.py:probe():196] Probing system
2023-11-18 12:32:19,229 DEBUG   HandlerThread:39554 [gitlib.py:_init_repo():56] git repository is invalid
2023-11-18 12:32:19,229 DEBUG   HandlerThread:39554 [system_info.py:probe():244] Probing system done
2023-11-18 12:32:19,230 DEBUG   HandlerThread:39554 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T10:32:19.139216', 'startedAt': '2023-11-18T10:32:18.452354', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=1', '--per_device_eval_batch_size=1', '--gradient_accumulation_steps=2', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.41075134277344}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}}
2023-11-18 12:32:19,230 INFO    HandlerThread:39554 [system_monitor.py:probe():224] Finished collecting system info
2023-11-18 12:32:19,230 INFO    HandlerThread:39554 [system_monitor.py:probe():227] Publishing system info
2023-11-18 12:32:19,230 DEBUG   HandlerThread:39554 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2023-11-18 12:32:19,232 DEBUG   HandlerThread:39554 [system_info.py:_save_pip():68] Saving pip packages done
2023-11-18 12:32:19,232 DEBUG   HandlerThread:39554 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2023-11-18 12:32:20,033 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/requirements.txt
2023-11-18 12:32:20,034 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/conda-environment.yaml
2023-11-18 12:32:22,038 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/conda-environment.yaml
2023-11-18 12:32:22,039 DEBUG   HandlerThread:39554 [system_info.py:_save_conda():87] Saving conda packages done
2023-11-18 12:32:22,041 INFO    HandlerThread:39554 [system_monitor.py:probe():229] Finished publishing system info
2023-11-18 12:32:22,045 DEBUG   SenderThread:39554 [sender.py:send():380] send: files
2023-11-18 12:32:22,045 INFO    SenderThread:39554 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now
2023-11-18 12:32:22,052 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:32:22,053 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:32:22,055 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:32:22,221 DEBUG   SenderThread:39554 [sender.py:send():380] send: telemetry
2023-11-18 12:32:22,221 DEBUG   SenderThread:39554 [sender.py:send():380] send: config
2023-11-18 12:32:22,222 DEBUG   SenderThread:39554 [sender.py:send():380] send: metric
2023-11-18 12:32:22,222 DEBUG   SenderThread:39554 [sender.py:send():380] send: telemetry
2023-11-18 12:32:22,222 DEBUG   SenderThread:39554 [sender.py:send():380] send: metric
2023-11-18 12:32:22,223 WARNING SenderThread:39554 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen)
2023-11-18 12:32:22,223 DEBUG   SenderThread:39554 [sender.py:send():380] send: telemetry
2023-11-18 12:32:22,658 INFO    wandb-upload_0:39554 [upload_job.py:push():131] Uploaded file /tmp/tmpnmps1g75wandb/3bw550az-wandb-metadata.json
2023-11-18 12:32:23,042 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/wandb-metadata.json
2023-11-18 12:32:23,043 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:32:23,636 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:32:25,050 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:32:27,151 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:32:28,799 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:32:33,800 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:32:37,052 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:32:37,053 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:32:37,054 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:32:39,241 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:32:44,242 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:32:49,245 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:32:50,246 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/config.yaml
2023-11-18 12:32:52,053 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:32:52,053 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:32:52,105 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:32:55,307 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:00,307 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:05,308 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:07,053 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:33:07,053 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:33:07,095 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:33:11,243 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:16,244 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:19,131 DEBUG   SystemMonitor:39554 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2023-11-18 12:33:19,132 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:33:22,053 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:33:22,054 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:33:22,095 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:33:22,249 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:27,250 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:32,250 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:37,053 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:33:37,054 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:33:37,095 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:33:38,214 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:43,215 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:48,216 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:49,133 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:33:52,053 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:33:52,054 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:33:52,095 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:33:53,313 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:33:58,314 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:03,315 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:07,053 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:34:07,054 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:34:07,095 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:34:09,288 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:14,290 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:19,133 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:34:20,134 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:22,054 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:34:22,054 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:34:22,057 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:34:22,621 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:34:24,629 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:34:25,332 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:30,333 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:35,334 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:37,054 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:34:37,054 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:34:37,095 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:34:41,269 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:46,270 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:49,134 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:34:52,056 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:34:52,056 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:34:52,057 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:34:52,057 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:34:57,312 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:02,313 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:07,055 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:35:08,167 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:08,167 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:35:08,168 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:35:12,462 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:18,404 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:18,775 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:35:20,025 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:35:22,264 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:35:22,264 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:35:23,178 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:35:23,560 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:28,560 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:33,561 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:37,263 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:35:37,264 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:35:38,178 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:35:39,538 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:44,540 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:49,173 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:35:50,005 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:35:50,018 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:35:52,265 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:35:52,266 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:35:53,179 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:35:55,215 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:35:55,525 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:01,076 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:01,254 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:05,281 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:06,419 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:08,013 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:36:08,015 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:36:08,179 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:36:11,325 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:12,003 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:17,373 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:17,577 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:20,018 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:36:21,401 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:23,014 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:36:23,017 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:36:23,324 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:24,575 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:36:27,439 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:28,604 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:33,476 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:34,182 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:38,013 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:36:38,014 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:36:39,232 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:39,517 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:39,574 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:36:41,526 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:43,537 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:44,906 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:48,573 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:50,019 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:36:50,020 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:36:53,014 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:36:53,015 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:36:54,596 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:36:54,609 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:36:55,856 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:00,657 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:01,795 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:06,703 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:07,726 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:08,013 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:37:08,014 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:37:10,322 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:37:12,745 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:13,317 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:18,747 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:18,788 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:20,022 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:37:22,815 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:23,013 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:37:23,014 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:37:24,220 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:26,592 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:37:28,855 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:29,538 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:34,893 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:35,252 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:38,014 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:37:38,016 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:37:40,723 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:40,937 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:41,594 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:37:46,255 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:46,990 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:49,763 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: partial_history
2023-11-18 12:37:49,765 DEBUG   SenderThread:39554 [sender.py:send():380] send: metric
2023-11-18 12:37:49,766 DEBUG   SenderThread:39554 [sender.py:send():380] send: metric
2023-11-18 12:37:49,766 DEBUG   SenderThread:39554 [sender.py:send():380] send: metric
2023-11-18 12:37:49,766 DEBUG   SenderThread:39554 [sender.py:send():380] send: history
2023-11-18 12:37:49,766 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: summary_record
2023-11-18 12:37:49,781 INFO    SenderThread:39554 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 12:37:50,016 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/wandb-summary.json
2023-11-18 12:37:50,021 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:37:51,020 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:52,022 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:37:52,026 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:54,274 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:37:54,276 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:37:56,592 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:37:57,064 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:37:57,234 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:02,884 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:03,109 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:08,415 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:09,162 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:09,275 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:38:09,277 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:38:11,593 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:38:13,195 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:13,975 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:15,212 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/config.yaml
2023-11-18 12:38:19,243 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:19,528 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:20,032 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:38:24,274 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:38:24,274 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:38:25,299 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:25,424 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:27,645 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:38:30,645 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:31,346 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:35,373 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:36,109 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:39,275 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:38:39,276 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:38:41,415 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:41,752 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:42,645 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:38:47,464 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:47,717 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:50,023 DEBUG   SenderThread:39554 [sender.py:send():380] send: stats
2023-11-18 12:38:53,105 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:53,538 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:38:55,752 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:38:55,755 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status
2023-11-18 12:38:57,645 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 12:38:58,775 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:38:59,601 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:39:04,384 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:39:05,645 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:39:09,673 INFO    Thread-12 :39554 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_123218-m1nos8ra/files/output.log
2023-11-18 12:39:10,194 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 12:39:10,751 DEBUG   HandlerThread:39554 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 12:39:10,752 DEBUG   SenderThread:39554 [sender.py:send_request():407] send_request: stop_status