hf_eng_fra_trans / myerrors_1542.out
aatherton2024's picture
Training in progress epoch 0
413f656
0%| | 0/329 [00:00<?, ?it/s] 1%| | 2/329 [00:01<04:19, 1.26it/s] 1%| | 3/329 [00:03<06:32, 1.20s/it] 1%| | 4/329 [00:05<08:33, 1.58s/it] 2%|▏ | 5/329 [00:07<09:47, 1.81s/it] 2%|▏ | 6/329 [00:10<10:36, 1.97s/it] 2%|▏ | 7/329 [00:12<11:11, 2.09s/it] 2%|▏ | 8/329 [00:13<09:59, 1.87s/it] 3%|β–Ž | 9/329 [00:15<09:03, 1.70s/it] 3%|β–Ž | 10/329 [00:17<10:02, 1.89s/it] 3%|β–Ž | 11/329 [00:19<10:23, 1.96s/it] 4%|β–Ž | 12/329 [00:21<11:01, 2.09s/it] 4%|▍ | 13/329 [00:23<10:42, 2.03s/it] 4%|▍ | 14/329 [00:26<10:53, 2.07s/it] 5%|▍ | 15/329 [00:27<09:38, 1.84s/it] 5%|▍ | 16/329 [00:28<08:35, 1.65s/it] 5%|β–Œ | 17/329 [00:30<09:46, 1.88s/it] 5%|β–Œ | 18/329 [00:33<10:21, 2.00s/it] 6%|β–Œ | 19/329 [00:35<10:42, 2.07s/it] 6%|β–Œ | 20/329 [00:37<10:59, 2.14s/it] 6%|β–‹ | 21/329 [00:40<11:21, 2.21s/it] 7%|β–‹ | 22/329 [00:42<10:52, 2.13s/it] 7%|β–‹ | 23/329 [00:44<10:59, 2.16s/it] 7%|β–‹ | 24/329 [00:46<11:03, 2.17s/it] 8%|β–Š | 25/329 [00:48<10:41, 2.11s/it] 8%|β–Š | 26/329 [00:50<11:01, 2.18s/it] 8%|β–Š | 27/329 [00:53<11:05, 2.20s/it] 9%|β–Š | 28/329 [00:55<11:17, 2.25s/it] 9%|β–‰ | 29/329 [00:57<11:22, 2.27s/it] 9%|β–‰ | 30/329 [00:59<09:55, 1.99s/it] 9%|β–‰ | 31/329 [01:00<09:18, 1.87s/it] 10%|β–‰ | 32/329 [01:02<09:51, 1.99s/it] 10%|β–ˆ | 33/329 [01:05<10:15, 2.08s/it] 10%|β–ˆ | 34/329 [01:07<10:15, 2.09s/it] 11%|β–ˆ | 35/329 [01:09<10:28, 2.14s/it] 11%|β–ˆ | 36/329 [01:11<10:32, 2.16s/it] 11%|β–ˆ | 37/329 [01:13<09:45, 2.00s/it] 12%|β–ˆβ– | 38/329 [01:15<09:46, 2.02s/it] 12%|β–ˆβ– | 39/329 [01:16<08:37, 1.78s/it] 12%|β–ˆβ– | 40/329 [01:18<07:56, 1.65s/it] 12%|β–ˆβ– | 41/329 [01:20<08:40, 1.81s/it] 13%|β–ˆβ–Ž | 42/329 [01:22<09:24, 1.97s/it] 13%|β–ˆβ–Ž | 43/329 [01:24<09:45, 2.05s/it] 13%|β–ˆβ–Ž | 44/329 [01:27<10:12, 2.15s/it] 14%|β–ˆβ–Ž | 45/329 [01:29<10:28, 2.21s/it] 14%|β–ˆβ– | 46/329 [01:31<10:22, 2.20s/it] 14%|β–ˆβ– | 47/329 [01:33<10:08, 2.16s/it] 15%|β–ˆβ– | 48/329 [01:35<09:55, 2.12s/it] 15%|β–ˆβ– | 49/329 [01:38<10:02, 2.15s/it] 15%|β–ˆβ–Œ | 50/329 [01:40<10:07, 2.18s/it] 16%|β–ˆβ–Œ | 51/329 [01:42<10:11, 2.20s/it] 16%|β–ˆβ–Œ | 52/329 [01:44<10:21, 2.25s/it] 16%|β–ˆβ–Œ | 53/329 [01:47<10:13, 2.22s/it] 16%|β–ˆβ–‹ | 54/329 [01:48<08:43, 1.90s/it] 17%|β–ˆβ–‹ | 55/329 [01:50<09:10, 2.01s/it] 17%|β–ˆβ–‹ | 56/329 [01:52<09:07, 2.00s/it] 17%|β–ˆβ–‹ | 57/329 [01:54<09:21, 2.06s/it] 18%|β–ˆβ–Š | 58/329 [01:56<09:14, 2.05s/it] 18%|β–ˆβ–Š | 59/329 [01:58<09:29, 2.11s/it] 18%|β–ˆβ–Š | 60/329 [02:01<09:48, 2.19s/it] 19%|β–ˆβ–Š | 61/329 [02:03<10:00, 2.24s/it] 19%|β–ˆβ–‰ | 62/329 [02:05<09:48, 2.20s/it] 19%|β–ˆβ–‰ | 63/329 [02:08<09:54, 2.24s/it] 19%|β–ˆβ–‰ | 64/329 [02:10<09:30, 2.15s/it] 20%|β–ˆβ–‰ | 65/329 [02:12<09:37, 2.19s/it] 20%|β–ˆβ–ˆ | 66/329 [02:14<09:36, 2.19s/it] 20%|β–ˆβ–ˆ | 67/329 [02:16<09:20, 2.14s/it] 21%|β–ˆβ–ˆ | 68/329 [02:18<09:28, 2.18s/it] 21%|β–ˆβ–ˆ | 69/329 [02:21<09:39, 2.23s/it] 21%|β–ˆβ–ˆβ– | 70/329 [02:23<09:42, 2.25s/it] 22%|β–ˆβ–ˆβ– | 71/329 [02:25<09:41, 2.25s/it] 22%|β–ˆβ–ˆβ– | 72/329 [02:28<09:41, 2.26s/it] 22%|β–ˆβ–ˆβ– | 73/329 [02:29<09:15, 2.17s/it] 22%|β–ˆβ–ˆβ– | 74/329 [02:32<09:30, 2.24s/it] 23%|β–ˆβ–ˆβ–Ž | 75/329 [02:34<09:36, 2.27s/it] 23%|β–ˆβ–ˆβ–Ž | 76/329 [02:36<09:30, 2.26s/it] 23%|β–ˆβ–ˆβ–Ž | 77/329 [02:39<09:31, 2.27s/it] 24%|β–ˆβ–ˆβ–Ž | 78/329 [02:40<08:44, 2.09s/it] 24%|β–ˆβ–ˆβ– | 79/329 [02:42<08:37, 2.07s/it] 24%|β–ˆβ–ˆβ– | 80/329 [02:45<08:50, 2.13s/it] 25%|β–ˆβ–ˆβ– | 81/329 [02:47<09:13, 2.23s/it] 25%|β–ˆβ–ˆβ– | 82/329 [02:49<09:09, 2.23s/it] 25%|β–ˆβ–ˆβ–Œ | 83/329 [02:52<09:15, 2.26s/it] 26%|β–ˆβ–ˆβ–Œ | 84/329 [02:54<08:40, 2.12s/it] 26%|β–ˆβ–ˆβ–Œ | 85/329 [02:56<08:44, 2.15s/it] 26%|β–ˆβ–ˆβ–Œ | 86/329 [02:57<07:48, 1.93s/it] 26%|β–ˆβ–ˆβ–‹ | 87/329 [02:59<07:16, 1.81s/it] 27%|β–ˆβ–ˆβ–‹ | 88/329 [03:01<07:41, 1.92s/it] 27%|β–ˆβ–ˆβ–‹ | 89/329 [03:03<08:04, 2.02s/it] 27%|β–ˆβ–ˆβ–‹ | 90/329 [03:05<08:22, 2.10s/it] 28%|β–ˆβ–ˆβ–Š | 91/329 [03:08<08:28, 2.14s/it] 28%|β–ˆβ–ˆβ–Š | 92/329 [03:10<08:20, 2.11s/it] 28%|β–ˆβ–ˆβ–Š | 93/329 [03:12<08:41, 2.21s/it] 29%|β–ˆβ–ˆβ–Š | 94/329 [03:13<07:39, 1.96s/it] 29%|β–ˆβ–ˆβ–‰ | 95/329 [03:16<07:52, 2.02s/it] 29%|β–ˆβ–ˆβ–‰ | 96/329 [03:18<07:47, 2.01s/it] 29%|β–ˆβ–ˆβ–‰ | 97/329 [03:20<08:07, 2.10s/it] 30%|β–ˆβ–ˆβ–‰ | 98/329 [03:22<08:13, 2.14s/it] 30%|β–ˆβ–ˆβ–ˆ | 99/329 [03:24<08:22, 2.18s/it] 30%|β–ˆβ–ˆβ–ˆ | 100/329 [03:27<08:29, 2.22s/it] 31%|β–ˆβ–ˆβ–ˆ | 101/329 [03:28<07:34, 1.99s/it] 31%|β–ˆβ–ˆβ–ˆ | 102/329 [03:30<06:44, 1.78s/it] 31%|β–ˆβ–ˆβ–ˆβ– | 103/329 [03:32<07:09, 1.90s/it] 32%|β–ˆβ–ˆβ–ˆβ– | 104/329 [03:34<07:39, 2.04s/it] 32%|β–ˆβ–ˆβ–ˆβ– | 105/329 [03:36<07:00, 1.88s/it] 32%|β–ˆβ–ˆβ–ˆβ– | 106/329 [03:38<07:17, 1.96s/it] 33%|β–ˆβ–ˆβ–ˆβ–Ž | 107/329 [03:40<07:35, 2.05s/it] 33%|β–ˆβ–ˆβ–ˆβ–Ž | 108/329 [03:42<07:03, 1.92s/it] 33%|β–ˆβ–ˆβ–ˆβ–Ž | 109/329 [03:44<07:14, 1.98s/it] 33%|β–ˆβ–ˆβ–ˆβ–Ž | 110/329 [03:45<06:54, 1.89s/it] 34%|β–ˆβ–ˆβ–ˆβ–Ž | 111/329 [03:47<06:02, 1.66s/it] 34%|β–ˆβ–ˆβ–ˆβ– | 112/329 [03:49<06:36, 1.83s/it] 34%|β–ˆβ–ˆβ–ˆβ– | 113/329 [03:51<06:47, 1.89s/it] 35%|β–ˆβ–ˆβ–ˆβ– | 114/329 [03:52<06:13, 1.74s/it] 35%|β–ˆβ–ˆβ–ˆβ– | 115/329 [03:54<06:42, 1.88s/it] 35%|β–ˆβ–ˆβ–ˆβ–Œ | 116/329 [03:57<07:06, 2.00s/it] 36%|β–ˆβ–ˆβ–ˆβ–Œ | 117/329 [03:59<07:06, 2.01s/it] 36%|β–ˆβ–ˆβ–ˆβ–Œ | 118/329 [04:00<06:31, 1.85s/it] 36%|β–ˆβ–ˆβ–ˆβ–Œ | 119/329 [04:02<06:19, 1.81s/it] 36%|β–ˆβ–ˆβ–ˆβ–‹ | 120/329 [04:04<06:46, 1.94s/it] 37%|β–ˆβ–ˆβ–ˆβ–‹ | 121/329 [04:06<06:28, 1.87s/it] 37%|β–ˆβ–ˆβ–ˆβ–‹ | 122/329 [04:08<06:38, 1.92s/it] 37%|β–ˆβ–ˆβ–ˆβ–‹ | 123/329 [04:10<06:53, 2.01s/it] 38%|β–ˆβ–ˆβ–ˆβ–Š | 124/329 [04:12<07:01, 2.06s/it] 38%|β–ˆβ–ˆβ–ˆβ–Š | 125/329 [04:15<07:12, 2.12s/it] 38%|β–ˆβ–ˆβ–ˆβ–Š | 126/329 [04:17<07:15, 2.14s/it] 39%|β–ˆβ–ˆβ–ˆβ–Š | 127/329 [04:19<07:26, 2.21s/it] 39%|β–ˆβ–ˆβ–ˆβ–‰ | 128/329 [04:21<07:04, 2.11s/it] 39%|β–ˆβ–ˆβ–ˆβ–‰ | 129/329 [04:23<07:03, 2.12s/it]slurmstepd-dl: error: *** JOB 1542 ON dl CANCELLED AT 2023-09-17T15:47:56 ***