| 2022-01-30 23:59:27,435 INFO MainThread:2678958 [internal.py:wandb_internal():87] W&B internal server running at pid: 2678958, started at: 2022-01-30 23:59:27.435029 | |
| 2022-01-30 23:59:27,437 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: check_version | |
| 2022-01-30 23:59:27,437 INFO WriterThread:2678958 [datastore.py:open_for_write():77] open: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/run-2f5mzr6c.wandb | |
| 2022-01-30 23:59:27,440 DEBUG SenderThread:2678958 [sender.py:send():234] send: header | |
| 2022-01-30 23:59:27,440 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: check_version | |
| 2022-01-30 23:59:27,856 DEBUG SenderThread:2678958 [sender.py:send():234] send: run | |
| 2022-01-30 23:59:28,031 INFO SenderThread:2678958 [dir_watcher.py:__init__():169] watching files in: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files | |
| 2022-01-30 23:59:28,032 INFO SenderThread:2678958 [sender.py:_start_run_threads():804] run started: 2f5mzr6c with start time 1643605166 | |
| 2022-01-30 23:59:28,032 DEBUG SenderThread:2678958 [sender.py:send():234] send: summary | |
| 2022-01-30 23:59:28,032 INFO SenderThread:2678958 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 23:59:28,033 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: run_start | |
| 2022-01-30 23:59:28,040 DEBUG HandlerThread:2678958 [meta.py:__init__():40] meta init | |
| 2022-01-30 23:59:28,040 DEBUG HandlerThread:2678958 [meta.py:__init__():54] meta init done | |
| 2022-01-30 23:59:28,040 DEBUG HandlerThread:2678958 [meta.py:probe():214] probe | |
| 2022-01-30 23:59:28,046 DEBUG HandlerThread:2678958 [meta.py:_setup_git():204] setup git | |
| 2022-01-30 23:59:28,060 DEBUG HandlerThread:2678958 [meta.py:_setup_git():211] setup git done | |
| 2022-01-30 23:59:28,060 DEBUG HandlerThread:2678958 [meta.py:_save_pip():58] save pip | |
| 2022-01-30 23:59:28,060 DEBUG HandlerThread:2678958 [meta.py:_save_pip():72] save pip done | |
| 2022-01-30 23:59:28,060 DEBUG HandlerThread:2678958 [meta.py:probe():252] probe done | |
| 2022-01-30 23:59:28,063 DEBUG SenderThread:2678958 [sender.py:send():234] send: files | |
| 2022-01-30 23:59:28,063 INFO SenderThread:2678958 [sender.py:_save_file():939] saving file wandb-metadata.json with policy now | |
| 2022-01-30 23:59:28,071 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: stop_status | |
| 2022-01-30 23:59:28,071 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: stop_status | |
| 2022-01-30 23:59:28,199 DEBUG SenderThread:2678958 [sender.py:send():234] send: config | |
| 2022-01-30 23:59:28,200 DEBUG SenderThread:2678958 [sender.py:send():234] send: metric | |
| 2022-01-30 23:59:28,200 DEBUG SenderThread:2678958 [sender.py:send():234] send: metric | |
| 2022-01-30 23:59:28,200 WARNING SenderThread:2678958 [sender.py:send_metric():897] Seen metric with glob (shouldnt happen) | |
| 2022-01-30 23:59:28,627 INFO Thread-11 :2678958 [upload_job.py:push():137] Uploaded file /tmp/tmpp6bl39oswandb/6kg16zdm-wandb-metadata.json | |
| 2022-01-30 23:59:29,033 INFO Thread-8 :2678958 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/wandb-summary.json | |
| 2022-01-30 23:59:29,033 INFO Thread-8 :2678958 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/wandb-metadata.json | |
| 2022-01-30 23:59:29,033 INFO Thread-8 :2678958 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/output.log | |
| 2022-01-30 23:59:29,033 INFO Thread-8 :2678958 [dir_watcher.py:_on_file_created():217] file/dir created: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/requirements.txt | |
| 2022-01-30 23:59:31,032 INFO Thread-8 :2678958 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/output.log | |
| 2022-01-30 23:59:34,715 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:34,715 DEBUG SenderThread:2678958 [sender.py:send():234] send: telemetry | |
| 2022-01-30 23:59:34,716 DEBUG SenderThread:2678958 [sender.py:send():234] send: exit | |
| 2022-01-30 23:59:34,716 INFO SenderThread:2678958 [sender.py:send_exit():366] handling exit code: 1 | |
| 2022-01-30 23:59:34,716 INFO SenderThread:2678958 [sender.py:send_exit():368] handling runtime: 6 | |
| 2022-01-30 23:59:34,754 INFO SenderThread:2678958 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 23:59:34,754 INFO SenderThread:2678958 [sender.py:send_exit():374] send defer | |
| 2022-01-30 23:59:34,755 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:34,755 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:34,755 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 0 | |
| 2022-01-30 23:59:34,756 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:34,756 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 0 | |
| 2022-01-30 23:59:34,756 INFO SenderThread:2678958 [sender.py:transition_state():387] send defer: 1 | |
| 2022-01-30 23:59:34,756 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:34,757 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 1 | |
| 2022-01-30 23:59:34,880 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:34,880 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 1 | |
| 2022-01-30 23:59:34,880 INFO SenderThread:2678958 [sender.py:transition_state():387] send defer: 2 | |
| 2022-01-30 23:59:34,880 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:34,881 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:34,881 DEBUG SenderThread:2678958 [sender.py:send():234] send: stats | |
| 2022-01-30 23:59:34,882 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:34,882 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 2 | |
| 2022-01-30 23:59:34,882 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:34,883 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 2 | |
| 2022-01-30 23:59:34,883 INFO SenderThread:2678958 [sender.py:transition_state():387] send defer: 3 | |
| 2022-01-30 23:59:34,883 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:34,883 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 3 | |
| 2022-01-30 23:59:34,883 DEBUG SenderThread:2678958 [sender.py:send():234] send: summary | |
| 2022-01-30 23:59:34,896 INFO SenderThread:2678958 [sender.py:_save_file():939] saving file wandb-summary.json with policy end | |
| 2022-01-30 23:59:34,896 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:34,896 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 3 | |
| 2022-01-30 23:59:34,896 INFO SenderThread:2678958 [sender.py:transition_state():387] send defer: 4 | |
| 2022-01-30 23:59:34,897 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:34,897 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 4 | |
| 2022-01-30 23:59:34,897 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:34,897 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 4 | |
| 2022-01-30 23:59:34,984 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:35,034 INFO Thread-8 :2678958 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/wandb-summary.json | |
| 2022-01-30 23:59:35,034 INFO Thread-8 :2678958 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/output.log | |
| 2022-01-30 23:59:35,172 INFO SenderThread:2678958 [sender.py:transition_state():387] send defer: 5 | |
| 2022-01-30 23:59:35,173 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:35,173 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:35,173 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 5 | |
| 2022-01-30 23:59:35,173 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:35,173 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 5 | |
| 2022-01-30 23:59:35,173 INFO SenderThread:2678958 [dir_watcher.py:finish():283] shutting down directory watcher | |
| 2022-01-30 23:59:35,275 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,034 INFO Thread-8 :2678958 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/config.yaml | |
| 2022-01-30 23:59:36,035 INFO SenderThread:2678958 [dir_watcher.py:finish():313] scan: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files | |
| 2022-01-30 23:59:36,035 INFO SenderThread:2678958 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/config.yaml config.yaml | |
| 2022-01-30 23:59:36,035 INFO SenderThread:2678958 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/wandb-summary.json wandb-summary.json | |
| 2022-01-30 23:59:36,035 INFO SenderThread:2678958 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/requirements.txt requirements.txt | |
| 2022-01-30 23:59:36,036 INFO SenderThread:2678958 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/wandb-metadata.json wandb-metadata.json | |
| 2022-01-30 23:59:36,036 INFO SenderThread:2678958 [dir_watcher.py:finish():327] scan save: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/output.log output.log | |
| 2022-01-30 23:59:36,036 INFO SenderThread:2678958 [sender.py:transition_state():387] send defer: 6 | |
| 2022-01-30 23:59:36,036 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,042 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:36,043 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 6 | |
| 2022-01-30 23:59:36,048 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:36,048 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 6 | |
| 2022-01-30 23:59:36,048 INFO SenderThread:2678958 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-30 23:59:36,144 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,144 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,246 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,247 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,349 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,349 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,451 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,452 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,554 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,554 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,585 INFO Thread-12 :2678958 [upload_job.py:push():137] Uploaded file /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/config.yaml | |
| 2022-01-30 23:59:36,593 INFO Thread-14 :2678958 [upload_job.py:push():137] Uploaded file /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/requirements.txt | |
| 2022-01-30 23:59:36,600 INFO Thread-15 :2678958 [upload_job.py:push():137] Uploaded file /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/output.log | |
| 2022-01-30 23:59:36,609 INFO Thread-13 :2678958 [upload_job.py:push():137] Uploaded file /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/files/wandb-summary.json | |
| 2022-01-30 23:59:36,656 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,657 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,758 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,759 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,810 INFO Thread-7 :2678958 [sender.py:transition_state():387] send defer: 7 | |
| 2022-01-30 23:59:36,810 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:36,810 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 7 | |
| 2022-01-30 23:59:36,811 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:36,811 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 7 | |
| 2022-01-30 23:59:36,860 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:36,965 INFO SenderThread:2678958 [sender.py:transition_state():387] send defer: 8 | |
| 2022-01-30 23:59:36,966 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:36,966 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:36,966 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 8 | |
| 2022-01-30 23:59:36,967 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:36,967 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 8 | |
| 2022-01-30 23:59:36,967 INFO SenderThread:2678958 [sender.py:transition_state():387] send defer: 9 | |
| 2022-01-30 23:59:36,967 DEBUG SenderThread:2678958 [sender.py:send():234] send: final | |
| 2022-01-30 23:59:36,968 DEBUG SenderThread:2678958 [sender.py:send():234] send: footer | |
| 2022-01-30 23:59:36,968 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: defer | |
| 2022-01-30 23:59:36,968 INFO HandlerThread:2678958 [handler.py:handle_request_defer():147] handle defer: 9 | |
| 2022-01-30 23:59:36,968 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: defer | |
| 2022-01-30 23:59:36,968 INFO SenderThread:2678958 [sender.py:send_request_defer():383] handle sender defer: 9 | |
| 2022-01-30 23:59:37,067 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: poll_exit | |
| 2022-01-30 23:59:37,068 DEBUG SenderThread:2678958 [sender.py:send_request():248] send_request: poll_exit | |
| 2022-01-30 23:59:37,068 INFO SenderThread:2678958 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-30 23:59:37,319 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: get_summary | |
| 2022-01-30 23:59:37,319 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: sampled_history | |
| 2022-01-30 23:59:37,320 DEBUG HandlerThread:2678958 [handler.py:handle_request():130] handle_request: shutdown | |
| 2022-01-30 23:59:37,320 INFO HandlerThread:2678958 [handler.py:finish():731] shutting down handler | |
| 2022-01-30 23:59:37,968 INFO WriterThread:2678958 [datastore.py:close():281] close: /home/sagrilaft/Project/audio/xls-r-et/wandb/run-20220130_235926-2f5mzr6c/run-2f5mzr6c.wandb | |
| 2022-01-30 23:59:38,317 INFO SenderThread:2678958 [sender.py:finish():1070] shutting down sender | |
| 2022-01-30 23:59:38,317 INFO SenderThread:2678958 [file_pusher.py:finish():177] shutting down file pusher | |
| 2022-01-30 23:59:38,317 INFO SenderThread:2678958 [file_pusher.py:join():182] waiting for file pusher | |
| 2022-01-30 23:59:38,319 INFO MainThread:2678958 [internal.py:handle_exit():77] Internal process exited | |