Training in progress, step 1500

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4853d0216f7fa258d87bc7419ec7d31398e4e620f56a3d32e767690755ce7af3
+oid sha256:cf7e3f5f04c6f8f239d7c52edf050af109a843d72183237787fc3438afd897c1
 size 1262075377

runs/Jan09_22-00-50_brutasse/events.out.tfevents.1641765760.brutasse.31164.0

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5452cd998f77c6f5a9102c308056d3e6e70fe3e86852ac3d3d881e546aabab06
-size 6861
+oid sha256:4dd4b35898cffe3895a85b9924d08e49287cc7b3224beebb236d07417952570d
+size 7964

wandb/run-20220109_220240-1g372i3v/files/output.log

@@ -775,3 +775,395 @@ Configuration saved in ./preprocessor_config.json
 Configuration saved in ./checkpoint-1000/config.json
 Model weights saved in ./checkpoint-1000/pytorch_model.bin
 Configuration saved in ./checkpoint-1000/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+/home/patrick/python_bin/transformers/models/wav2vec2/modeling_wav2vec2.py:1107: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
+  return (input_length - kernel_size) // stride + 1
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  6%|███▎                                               | 1099/17200 [28:45<6:27:30,  1.44s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  7%|███▌                                               | 1199/17200 [30:50<6:23:22,  1.44s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  8%|███▊                                               | 1300/17200 [32:55<6:04:09,  1.37s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  8%|████▏                                              | 1398/17200 [34:57<5:07:16,  1.17s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  9%|████▍                                              | 1499/17200 [37:01<4:39:52,  1.07s/it]
+  9%|████▍                                              | 1500/17200 [37:02<4:21:20,  1.00it/s]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 4620
+  Batch size = 8
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+100%|████████████████████████████████████████████████████████| 578/578 [02:19<00:00,  4.34it/s]
+
+Configuration saved in ./checkpoint-1500/config.json
+Model weights saved in ./checkpoint-1500/pytorch_model.bin
+Configuration saved in ./checkpoint-1500/preprocessor_config.json

wandb/run-20220109_220240-1g372i3v/logs/debug-internal.log

@@ -2305,3 +2305,548 @@
 2022-01-09 22:29:14,744 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
 2022-01-09 22:29:14,745 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
 2022-01-09 22:29:23,634 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:29:28,956 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:29,857 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:29:29,857 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:29:30,957 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:32,957 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:34,957 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:36,958 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:38,958 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:40,958 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:42,959 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:44,926 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:29:44,927 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:29:44,959 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:46,959 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:48,959 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:50,960 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:52,960 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:53,904 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:29:54,960 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:56,961 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:29:58,961 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:00,002 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:30:00,002 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:30:01,962 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:03,962 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:05,962 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:07,962 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:09,963 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:11,963 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:13,963 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:15,080 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:30:15,080 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:30:15,964 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:17,964 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:19,964 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:21,965 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:23,965 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:24,160 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:30:25,965 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:27,966 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:29,966 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:30,239 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:30:30,239 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:30:31,966 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:33,967 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:35,967 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:37,967 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:39,968 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:41,968 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:43,968 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:45,319 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:30:45,319 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:30:45,969 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:47,969 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:49,969 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:51,970 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:53,970 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:54,407 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:30:55,970 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:57,970 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:30:59,971 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:00,391 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:31:00,391 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:31:01,971 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:03,971 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:05,972 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:07,972 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:09,972 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:11,973 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:13,973 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:15,470 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:31:15,471 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:31:15,973 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:17,974 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:19,974 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:21,974 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:23,975 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:24,656 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:31:25,975 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:27,975 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:29,976 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:30,558 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:31:30,558 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:31:31,976 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:33,488 DEBUG   SenderThread:31514 [sender.py:send():182] send: history
+2022-01-09 22:31:33,537 DEBUG   SenderThread:31514 [sender.py:send():182] send: summary
+2022-01-09 22:31:33,561 INFO    SenderThread:31514 [sender.py:_save_file():873] saving file wandb-summary.json with policy end
+2022-01-09 22:31:33,977 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/wandb-summary.json
+2022-01-09 22:31:33,977 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:35,977 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:37,977 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:39,977 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:41,978 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:43,978 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:45,695 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:31:45,696 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:31:45,978 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:47,979 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:49,979 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:51,979 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:53,980 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:54,910 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:31:55,980 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:57,980 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:31:59,981 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:00,790 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:32:00,791 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:32:01,981 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:03,982 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:05,982 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:07,982 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:10,983 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:12,983 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:14,984 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:15,875 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:32:15,875 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:32:16,984 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:18,984 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:20,984 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:22,985 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:24,985 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:25,155 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:32:26,985 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:28,986 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:30,959 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:32:30,959 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:32:30,986 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:32,986 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:34,987 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:36,987 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:38,987 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:40,988 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:42,988 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:44,988 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:46,075 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:32:46,075 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:32:46,989 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:48,989 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:50,989 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:52,989 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:54,990 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:55,396 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:32:56,990 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:32:58,990 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:00,991 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:01,181 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:33:01,181 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:33:02,991 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:04,991 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:06,992 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:08,992 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:10,992 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:12,993 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:14,993 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:16,308 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:33:16,309 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:33:16,993 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:18,994 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:20,994 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:22,994 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:24,995 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:25,614 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:33:26,995 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:28,995 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:30,996 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:31,453 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:33:31,453 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:33:32,996 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:34,997 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:36,997 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:38,455 DEBUG   SenderThread:31514 [sender.py:send():182] send: history
+2022-01-09 22:33:38,508 DEBUG   SenderThread:31514 [sender.py:send():182] send: summary
+2022-01-09 22:33:38,530 INFO    SenderThread:31514 [sender.py:_save_file():873] saving file wandb-summary.json with policy end
+2022-01-09 22:33:38,997 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/wandb-summary.json
+2022-01-09 22:33:38,998 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:39,998 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:40,998 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:42,998 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:44,998 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:46,569 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:33:46,570 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:33:46,999 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:48,999 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:50,999 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:53,000 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:55,000 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:55,840 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:33:57,000 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:33:59,001 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:01,001 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:01,661 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:34:01,661 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:34:03,001 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:05,002 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:07,002 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:09,002 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:11,002 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:13,003 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:15,003 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:16,823 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:34:16,823 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:34:17,003 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:19,004 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:21,004 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:23,004 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:25,005 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:26,068 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:34:28,005 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:30,005 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:31,937 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:34:31,937 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:34:32,007 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:34,007 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:36,007 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:38,008 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:40,008 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:42,008 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:44,009 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:46,009 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:47,007 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:34:47,007 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:34:48,009 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:50,010 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:52,010 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:54,010 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:56,010 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:34:56,290 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:34:58,011 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:00,011 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:02,011 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:02,086 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:35:02,086 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:35:04,012 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:06,012 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:08,012 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:10,013 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:12,013 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:14,013 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:16,013 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:17,175 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:35:17,175 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:35:18,014 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:20,014 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:22,014 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:24,015 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:26,015 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:26,518 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:35:28,015 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:30,016 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:32,016 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:32,258 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:35:32,258 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:35:34,016 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:36,017 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:38,017 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:40,017 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:42,017 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:42,207 DEBUG   SenderThread:31514 [sender.py:send():182] send: history
+2022-01-09 22:35:42,250 DEBUG   SenderThread:31514 [sender.py:send():182] send: summary
+2022-01-09 22:35:42,272 INFO    SenderThread:31514 [sender.py:_save_file():873] saving file wandb-summary.json with policy end
+2022-01-09 22:35:43,018 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/wandb-summary.json
+2022-01-09 22:35:44,018 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:46,018 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:47,333 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:35:47,333 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:35:48,018 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:50,019 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:54,019 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:56,020 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:35:56,739 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:35:58,020 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:00,020 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:02,021 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:02,406 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:36:02,406 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:36:04,021 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:06,021 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:08,022 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:10,022 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:12,022 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:14,023 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:16,023 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:17,492 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:36:17,493 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:36:18,023 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:20,024 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:22,024 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:24,024 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:26,024 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:26,963 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:36:28,025 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:30,025 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:32,025 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:32,569 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:36:32,569 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:36:34,026 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:36,026 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:38,026 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:40,027 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:42,027 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:44,027 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:46,027 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:47,673 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:36:47,673 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:36:48,028 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:50,028 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:52,029 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:55,029 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:57,029 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:36:57,190 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:36:59,030 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:01,030 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:02,771 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:37:02,771 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:37:03,030 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:05,031 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:07,031 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:09,031 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:11,031 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:13,032 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:15,032 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:17,032 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:17,881 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:37:17,881 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:37:19,033 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:21,033 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:23,033 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:25,034 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:27,034 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:27,415 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:37:29,034 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:31,035 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:32,964 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:37:32,965 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:37:33,035 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:35,035 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:37,036 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:39,036 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:41,036 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:43,036 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:45,037 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:45,875 DEBUG   SenderThread:31514 [sender.py:send():182] send: history
+2022-01-09 22:37:45,918 DEBUG   SenderThread:31514 [sender.py:send():182] send: summary
+2022-01-09 22:37:45,942 INFO    SenderThread:31514 [sender.py:_save_file():873] saving file wandb-summary.json with policy end
+2022-01-09 22:37:46,037 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/wandb-summary.json
+2022-01-09 22:37:47,037 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:48,120 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:37:48,121 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:37:49,038 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:51,038 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:53,038 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:55,038 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:57,039 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:37:57,642 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:37:59,039 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:01,039 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:03,040 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:03,201 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:38:03,201 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:38:05,040 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:07,040 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:09,041 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:11,041 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:13,041 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:15,042 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:17,042 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:18,302 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:38:18,302 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:38:19,042 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:21,043 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:23,043 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:25,043 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:27,044 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:27,872 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:38:29,044 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:31,044 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:33,045 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:33,422 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:38:33,422 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:38:35,045 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:37,045 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:39,046 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:41,046 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:43,046 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:45,047 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:47,047 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:48,495 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:38:48,495 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:38:49,047 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:51,048 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:53,048 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:55,048 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:57,049 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:38:58,083 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:38:59,049 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:01,049 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:03,049 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:03,578 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:39:03,579 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:39:06,050 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:08,050 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:10,051 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:12,051 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:14,051 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:16,051 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:18,052 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:18,681 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:39:18,682 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:39:20,052 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:22,052 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:24,053 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:26,053 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:28,053 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:28,303 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:39:30,053 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:32,054 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:33,777 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:39:33,777 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:39:34,054 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:36,055 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:38,055 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:40,055 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:42,056 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:44,056 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:46,056 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:48,057 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:48,869 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:39:48,869 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:39:49,455 DEBUG   SenderThread:31514 [sender.py:send():182] send: history
+2022-01-09 22:39:49,497 DEBUG   SenderThread:31514 [sender.py:send():182] send: summary
+2022-01-09 22:39:49,519 INFO    SenderThread:31514 [sender.py:_save_file():873] saving file wandb-summary.json with policy end
+2022-01-09 22:39:50,057 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/wandb-summary.json
+2022-01-09 22:39:50,057 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:51,057 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:52,057 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:54,058 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:56,058 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:58,058 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:39:58,535 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:40:00,059 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:02,059 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:03,943 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:40:03,943 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:40:04,059 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:06,060 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:08,060 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:10,060 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:12,061 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:14,061 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:16,061 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:18,062 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:19,039 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:40:19,039 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:40:20,062 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:22,062 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:24,063 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:26,063 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:28,063 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:28,773 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:40:30,063 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:32,064 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:34,064 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:34,136 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:40:34,136 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:40:36,064 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:38,065 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:40,065 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:42,065 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:44,066 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:46,066 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:48,066 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:49,257 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:40:49,257 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:40:50,067 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:52,067 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:54,067 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:56,068 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:58,068 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:40:59,004 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:41:00,068 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:02,068 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:04,069 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:04,413 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:41:04,413 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:41:06,069 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:08,069 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:10,070 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:12,070 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:15,071 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:17,071 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:19,071 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:19,528 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:41:19,528 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:41:21,072 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:23,072 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:25,072 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:27,073 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:29,073 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:29,240 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:41:31,073 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:33,073 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:34,634 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:41:34,634 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:41:35,074 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:37,074 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:39,074 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:41,075 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:43,075 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:45,075 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:47,076 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:49,076 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:49,709 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:41:49,709 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:41:51,076 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:53,077 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:55,077 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:57,077 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:59,078 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:41:59,481 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:42:01,078 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:03,078 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:04,846 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:42:04,846 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:42:05,079 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:07,079 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:09,079 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:11,080 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:11,408 DEBUG   SenderThread:31514 [sender.py:send():182] send: history
+2022-01-09 22:42:11,421 DEBUG   SenderThread:31514 [sender.py:send():182] send: summary
+2022-01-09 22:42:11,442 INFO    SenderThread:31514 [sender.py:_save_file():873] saving file wandb-summary.json with policy end
+2022-01-09 22:42:12,080 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/wandb-summary.json
+2022-01-09 22:42:12,080 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:13,080 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:15,081 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:19,920 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:42:19,921 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:42:29,768 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:42:34,999 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:42:34,999 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:42:47,094 INFO    Thread-8  :31514 [dir_watcher.py:_on_file_modified():230] file/dir modified: /home/patrick/hugging_face/examples/xls-r-300m-sv/wandb/run-20220109_220240-1g372i3v/files/output.log
+2022-01-09 22:42:50,080 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:42:50,080 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status
+2022-01-09 22:43:00,077 DEBUG   SenderThread:31514 [sender.py:send():182] send: stats
+2022-01-09 22:43:05,154 DEBUG   HandlerThread:31514 [handler.py:handle_request():126] handle_request: stop_status
+2022-01-09 22:43:05,155 DEBUG   SenderThread:31514 [sender.py:send_request():196] send_request: stop_status