Training in progress, step 1500

.gitattributes

@@ -25,3 +25,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zstandard filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+wandb/run-20220202_014046-gq6zg0t4/run-gq6zg0t4.wandb filter=lfs diff=lfs merge=lfs -text

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f2d18eede7a3ae0b6020edc089f2eed535af1939d98effa33b6222d14d111346
+oid sha256:9e68dab267e6ae49ebd9851f76298bd82930c1faa353cae9d1731fa5a25a33d1
 size 1262083569

wandb/run-20220202_014046-gq6zg0t4/files/output.log

@@ -1095,3 +1095,554 @@ Configuration saved in ./checkpoint-1000/config.json
 {'eval_loss': 0.4216161072254181, 'eval_wer': 0.4414151661171697, 'eval_runtime': 131.8019, 'eval_samples_per_second': 19.825, 'eval_steps_per_second': 0.281, 'epoch': 25.0}
 Model weights saved in ./checkpoint-1000/pytorch_model.bin
 Configuration saved in ./checkpoint-1000/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-500] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 28%|█████████████████████████████████████████████▍                                                                                                                       | 1100/4000 [2:40:48<7:57:31,  9.88s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 30%|█████████████████████████████████████████████████▍                                                                                                                   | 1199/4000 [2:54:36<6:15:16,  8.04s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 32%|█████████████████████████████████████████████████████▋                                                                                                               | 1300/4000 [3:08:53<7:22:27,  9.83s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 35%|█████████████████████████████████████████████████████████▋                                                                                                           | 1399/4000 [3:22:41<5:39:48,  7.84s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 38%|█████████████████████████████████████████████████████████████▉                                                                                                       | 1500/4000 [3:36:55<6:49:40,  9.83s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 2613
+  Batch size = 72
+{'loss': 0.1601, 'learning_rate': 0.0008120956604474414, 'epoch': 37.5}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+Configuration saved in ./checkpoint-1500/config.json
+{'eval_loss': 0.4490557909011841, 'eval_wer': 0.429527009890946, 'eval_runtime': 132.5526, 'eval_samples_per_second': 19.713, 'eval_steps_per_second': 0.279, 'epoch': 37.5}
+Model weights saved in ./checkpoint-1500/pytorch_model.bin
+Configuration saved in ./checkpoint-1500/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Adding files tracked by Git LFS: ['wandb/run-20220202_014046-gq6zg0t4/run-gq6zg0t4.wandb']. This may take a bit of time if the files are large.