Hugging Face's logo

Finnish-NLP
/
whisper-large-finnish-v3

Automatic Speech Recognition
Transformers
TensorBoard
Safetensors
Finnish
whisper
whisper-event
finnish
speech-recognition
Eval Results
Inference Endpoints
Model card Files Metrics Community
1
whisper-large-finnish-v3
File size: 30,510 Bytes 
b0ae254
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
 
2023-11-18 13:10:05,570 INFO    StreamThr :56156 [internal.py:wandb_internal():86] W&B internal server running at pid: 56156, started at: 2023-11-18 13:10:05.569504
2023-11-18 13:10:05,572 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status
2023-11-18 13:10:05,575 INFO    WriterThread:56156 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_131005-esux4gxo/run-esux4gxo.wandb
2023-11-18 13:10:05,577 DEBUG   SenderThread:56156 [sender.py:send():380] send: header
2023-11-18 13:10:05,577 DEBUG   SenderThread:56156 [sender.py:send():380] send: run
2023-11-18 13:10:05,992 INFO    SenderThread:56156 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_131005-esux4gxo/files
2023-11-18 13:10:05,992 INFO    SenderThread:56156 [sender.py:_start_run_threads():1124] run started: esux4gxo with start time 1700305805.57009
2023-11-18 13:10:06,003 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: check_version
2023-11-18 13:10:06,004 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: check_version
2023-11-18 13:10:06,058 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: run_start
2023-11-18 13:10:06,175 DEBUG   HandlerThread:56156 [system_info.py:__init__():32] System info init
2023-11-18 13:10:06,176 DEBUG   HandlerThread:56156 [system_info.py:__init__():47] System info init done
2023-11-18 13:10:06,176 INFO    HandlerThread:56156 [system_monitor.py:start():194] Starting system monitor
2023-11-18 13:10:06,176 INFO    SystemMonitor:56156 [system_monitor.py:_start():158] Starting system asset monitoring threads
2023-11-18 13:10:06,177 INFO    HandlerThread:56156 [system_monitor.py:probe():214] Collecting system info
2023-11-18 13:10:06,178 INFO    SystemMonitor:56156 [interfaces.py:start():190] Started cpu monitoring
2023-11-18 13:10:06,179 INFO    SystemMonitor:56156 [interfaces.py:start():190] Started disk monitoring
2023-11-18 13:10:06,180 INFO    SystemMonitor:56156 [interfaces.py:start():190] Started gpu monitoring
2023-11-18 13:10:06,181 INFO    SystemMonitor:56156 [interfaces.py:start():190] Started memory monitoring
2023-11-18 13:10:06,183 INFO    SystemMonitor:56156 [interfaces.py:start():190] Started network monitoring
2023-11-18 13:10:06,189 DEBUG   HandlerThread:56156 [system_info.py:probe():196] Probing system
2023-11-18 13:10:06,294 DEBUG   HandlerThread:56156 [gitlib.py:_init_repo():56] git repository is invalid
2023-11-18 13:10:06,295 DEBUG   HandlerThread:56156 [system_info.py:probe():244] Probing system done
2023-11-18 13:10:06,295 DEBUG   HandlerThread:56156 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T11:10:06.189833', 'startedAt': '2023-11-18T11:10:05.529442', 'docker': None, 'cuda': None, 'args': ('--local_rank=0', '--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v3', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=WhisperLargeV3Finnish', '--max_steps=60000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=2', '--logging_steps=20', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--bf16', '--torch_compile', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub', '--push_to_hub_model_id=WhisperLargeFinnishV3'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.435340881347656}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}}
2023-11-18 13:10:06,295 INFO    HandlerThread:56156 [system_monitor.py:probe():224] Finished collecting system info
2023-11-18 13:10:06,295 INFO    HandlerThread:56156 [system_monitor.py:probe():227] Publishing system info
2023-11-18 13:10:06,296 DEBUG   HandlerThread:56156 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2023-11-18 13:10:06,297 DEBUG   HandlerThread:56156 [system_info.py:_save_pip():68] Saving pip packages done
2023-11-18 13:10:06,298 DEBUG   HandlerThread:56156 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2023-11-18 13:10:06,996 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/conda-environment.yaml
2023-11-18 13:10:06,996 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/requirements.txt
2023-11-18 13:10:08,527 DEBUG   HandlerThread:56156 [system_info.py:_save_conda():87] Saving conda packages done
2023-11-18 13:10:08,539 INFO    HandlerThread:56156 [system_monitor.py:probe():229] Finished publishing system info
2023-11-18 13:10:08,542 DEBUG   SenderThread:56156 [sender.py:send():380] send: files
2023-11-18 13:10:08,543 INFO    SenderThread:56156 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now
2023-11-18 13:10:08,550 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 13:10:08,551 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: stop_status
2023-11-18 13:10:08,554 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:10:08,730 DEBUG   SenderThread:56156 [sender.py:send():380] send: telemetry
2023-11-18 13:10:08,731 DEBUG   SenderThread:56156 [sender.py:send():380] send: config
2023-11-18 13:10:08,732 DEBUG   SenderThread:56156 [sender.py:send():380] send: metric
2023-11-18 13:10:08,732 DEBUG   SenderThread:56156 [sender.py:send():380] send: telemetry
2023-11-18 13:10:08,732 DEBUG   SenderThread:56156 [sender.py:send():380] send: metric
2023-11-18 13:10:08,733 WARNING SenderThread:56156 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen)
2023-11-18 13:10:08,733 DEBUG   SenderThread:56156 [sender.py:send():380] send: telemetry
2023-11-18 13:10:09,003 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/conda-environment.yaml
2023-11-18 13:10:09,005 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log
2023-11-18 13:10:09,005 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-metadata.json
2023-11-18 13:10:10,150 INFO    wandb-upload_0:56156 [upload_job.py:push():131] Uploaded file /tmp/tmpum_ty6bjwandb/lr973oc6-wandb-metadata.json
2023-11-18 13:10:11,011 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log
2023-11-18 13:10:11,303 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:13,019 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log
2023-11-18 13:10:16,303 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:21,304 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:23,550 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 13:10:23,551 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: stop_status
2023-11-18 13:10:23,551 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:10:26,717 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:31,718 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:36,721 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:37,123 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/config.yaml
2023-11-18 13:10:38,551 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 13:10:38,551 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: stop_status
2023-11-18 13:10:38,605 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:10:42,797 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:47,798 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:52,799 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:10:53,551 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 13:10:53,552 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: stop_status
2023-11-18 13:10:53,595 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:10:57,815 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:02,816 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:06,183 DEBUG   SystemMonitor:56156 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2023-11-18 13:11:06,185 DEBUG   SenderThread:56156 [sender.py:send():380] send: stats
2023-11-18 13:11:08,186 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:08,551 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 13:11:08,552 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: stop_status
2023-11-18 13:11:08,595 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:11:13,806 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:18,808 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:23,551 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 13:11:23,552 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: stop_status
2023-11-18 13:11:23,595 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:11:23,815 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:28,816 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:33,817 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:36,186 DEBUG   SenderThread:56156 [sender.py:send():380] send: stats
2023-11-18 13:11:38,551 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 13:11:38,552 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: stop_status
2023-11-18 13:11:38,605 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:11:39,781 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:44,782 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:49,783 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:53,552 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 13:11:53,552 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: stop_status
2023-11-18 13:11:53,605 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:11:54,799 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:11:59,801 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:12:04,801 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:12:06,186 DEBUG   SenderThread:56156 [sender.py:send():380] send: stats
2023-11-18 13:12:06,293 DEBUG   SenderThread:56156 [sender.py:send():380] send: exit
2023-11-18 13:12:06,293 INFO    SenderThread:56156 [sender.py:send_exit():585] handling exit code: 1
2023-11-18 13:12:06,294 INFO    SenderThread:56156 [sender.py:send_exit():587] handling runtime: 120
2023-11-18 13:12:06,306 INFO    SenderThread:56156 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 13:12:06,308 INFO    SenderThread:56156 [sender.py:send_exit():593] send defer
2023-11-18 13:12:06,309 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,309 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 0
2023-11-18 13:12:06,310 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,310 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 0
2023-11-18 13:12:06,310 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 1
2023-11-18 13:12:06,310 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,311 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 1
2023-11-18 13:12:06,311 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,311 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 1
2023-11-18 13:12:06,312 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 2
2023-11-18 13:12:06,312 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,312 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 2
2023-11-18 13:12:06,312 INFO    HandlerThread:56156 [system_monitor.py:finish():203] Stopping system monitor
2023-11-18 13:12:06,313 INFO    HandlerThread:56156 [interfaces.py:finish():202] Joined cpu monitor
2023-11-18 13:12:06,313 DEBUG   SystemMonitor:56156 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2023-11-18 13:12:06,313 INFO    HandlerThread:56156 [interfaces.py:finish():202] Joined disk monitor
2023-11-18 13:12:06,314 DEBUG   SystemMonitor:56156 [system_monitor.py:_start():183] Publishing last batch of metrics
2023-11-18 13:12:06,325 INFO    HandlerThread:56156 [interfaces.py:finish():202] Joined gpu monitor
2023-11-18 13:12:06,325 INFO    HandlerThread:56156 [interfaces.py:finish():202] Joined memory monitor
2023-11-18 13:12:06,325 INFO    HandlerThread:56156 [interfaces.py:finish():202] Joined network monitor
2023-11-18 13:12:06,326 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,326 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 2
2023-11-18 13:12:06,326 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 3
2023-11-18 13:12:06,327 DEBUG   SenderThread:56156 [sender.py:send():380] send: stats
2023-11-18 13:12:06,327 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,327 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 3
2023-11-18 13:12:06,328 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,328 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 3
2023-11-18 13:12:06,328 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 4
2023-11-18 13:12:06,329 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,329 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 4
2023-11-18 13:12:06,330 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,330 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 4
2023-11-18 13:12:06,330 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 5
2023-11-18 13:12:06,331 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,331 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 5
2023-11-18 13:12:06,331 DEBUG   SenderThread:56156 [sender.py:send():380] send: summary
2023-11-18 13:12:06,333 INFO    SenderThread:56156 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 13:12:06,335 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,335 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 5
2023-11-18 13:12:06,335 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 6
2023-11-18 13:12:06,336 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,336 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 6
2023-11-18 13:12:06,337 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,337 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 6
2023-11-18 13:12:06,337 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 7
2023-11-18 13:12:06,337 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 13:12:06,338 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,338 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 7
2023-11-18 13:12:06,338 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,339 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 7
2023-11-18 13:12:06,523 INFO    Thread-12 :56156 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-summary.json
2023-11-18 13:12:06,872 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 8
2023-11-18 13:12:06,872 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,872 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 8
2023-11-18 13:12:06,873 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,873 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 8
2023-11-18 13:12:06,874 INFO    SenderThread:56156 [job_builder.py:build():298] Attempting to build job artifact
2023-11-18 13:12:06,880 INFO    SenderThread:56156 [job_builder.py:_get_source_type():439] no source found
2023-11-18 13:12:06,880 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 9
2023-11-18 13:12:06,881 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:06,881 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 9
2023-11-18 13:12:06,882 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:06,882 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 9
2023-11-18 13:12:06,882 INFO    SenderThread:56156 [dir_watcher.py:finish():358] shutting down directory watcher
2023-11-18 13:12:07,292 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 13:12:07,540 INFO    SenderThread:56156 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log
2023-11-18 13:12:07,541 INFO    SenderThread:56156 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_131005-esux4gxo/files
2023-11-18 13:12:07,543 INFO    SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/conda-environment.yaml conda-environment.yaml
2023-11-18 13:12:07,548 INFO    SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/config.yaml config.yaml
2023-11-18 13:12:07,553 INFO    SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log output.log
2023-11-18 13:12:07,558 INFO    SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/requirements.txt requirements.txt
2023-11-18 13:12:07,563 INFO    SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-metadata.json wandb-metadata.json
2023-11-18 13:12:07,564 INFO    SenderThread:56156 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-summary.json wandb-summary.json
2023-11-18 13:12:07,570 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 10
2023-11-18 13:12:07,570 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 13:12:07,571 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:07,572 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 10
2023-11-18 13:12:07,574 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:07,574 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 10
2023-11-18 13:12:07,575 INFO    SenderThread:56156 [file_pusher.py:finish():175] shutting down file pusher
2023-11-18 13:12:07,936 INFO    wandb-upload_1:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/config.yaml
2023-11-18 13:12:07,982 INFO    wandb-upload_0:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/conda-environment.yaml
2023-11-18 13:12:08,041 INFO    wandb-upload_2:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/output.log
2023-11-18 13:12:08,059 INFO    wandb-upload_3:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/requirements.txt
2023-11-18 13:12:08,059 INFO    wandb-upload_4:56156 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_131005-esux4gxo/files/wandb-summary.json
2023-11-18 13:12:08,260 INFO    Thread-11 (_thread_body):56156 [sender.py:transition_state():613] send defer: 11
2023-11-18 13:12:08,260 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:08,260 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 11
2023-11-18 13:12:08,261 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:08,261 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 11
2023-11-18 13:12:08,261 INFO    SenderThread:56156 [file_pusher.py:join():181] waiting for file pusher
2023-11-18 13:12:08,261 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 12
2023-11-18 13:12:08,262 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:08,262 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 12
2023-11-18 13:12:08,262 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:08,262 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 12
2023-11-18 13:12:08,263 INFO    SenderThread:56156 [file_stream.py:finish():595] file stream finish called
2023-11-18 13:12:08,292 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 13:12:08,775 INFO    SenderThread:56156 [file_stream.py:finish():599] file stream finish is done
2023-11-18 13:12:08,776 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 13
2023-11-18 13:12:08,776 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 13:12:08,776 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:08,777 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 13
2023-11-18 13:12:08,777 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:08,777 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 13
2023-11-18 13:12:08,778 INFO    SenderThread:56156 [sender.py:transition_state():613] send defer: 14
2023-11-18 13:12:08,778 DEBUG   SenderThread:56156 [sender.py:send():380] send: final
2023-11-18 13:12:08,778 DEBUG   SenderThread:56156 [sender.py:send():380] send: footer
2023-11-18 13:12:08,778 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: defer
2023-11-18 13:12:08,779 INFO    HandlerThread:56156 [handler.py:handle_request_defer():172] handle defer: 14
2023-11-18 13:12:08,780 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: defer
2023-11-18 13:12:08,780 INFO    SenderThread:56156 [sender.py:send_request_defer():609] handle sender defer: 14
2023-11-18 13:12:08,781 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 13:12:08,781 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 13:12:08,781 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 13:12:08,781 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: server_info
2023-11-18 13:12:08,782 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 13:12:08,782 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: get_summary
2023-11-18 13:12:08,782 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: server_info
2023-11-18 13:12:08,783 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: sampled_history
2023-11-18 13:12:08,784 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 13:12:08,785 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: job_info
2023-11-18 13:12:08,929 DEBUG   SenderThread:56156 [sender.py:send_request():407] send_request: job_info
2023-11-18 13:12:08,930 INFO    MainThread:56156 [wandb_run.py:_footer_history_summary_info():3760] rendering history
2023-11-18 13:12:08,930 INFO    MainThread:56156 [wandb_run.py:_footer_history_summary_info():3792] rendering summary
2023-11-18 13:12:08,931 INFO    MainThread:56156 [wandb_run.py:_footer_sync_info():3719] logging synced files
2023-11-18 13:12:08,931 DEBUG   HandlerThread:56156 [handler.py:handle_request():146] handle_request: shutdown
2023-11-18 13:12:08,932 INFO    HandlerThread:56156 [handler.py:finish():866] shutting down handler
2023-11-18 13:12:09,785 INFO    WriterThread:56156 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_131005-esux4gxo/run-esux4gxo.wandb
2023-11-18 13:12:09,930 INFO    SenderThread:56156 [sender.py:finish():1536] shutting down sender
2023-11-18 13:12:09,930 INFO    SenderThread:56156 [file_pusher.py:finish():175] shutting down file pusher
2023-11-18 13:12:09,931 INFO    SenderThread:56156 [file_pusher.py:join():181] waiting for file pusher