language_model.model.layers.0 2
language_model.model.layers.1 2
language_model.model.layers.2 2
language_model.model.layers.3 2
language_model.model.layers.4 2
language_model.model.layers.5 2
language_model.model.layers.6 2
language_model.model.layers.7 2
language_model.model.layers.8 2
language_model.model.layers.9 4
language_model.model.layers.10 4
language_model.model.layers.11 4
language_model.model.layers.12 4
language_model.model.layers.13 4
language_model.model.layers.14 4
language_model.model.layers.15 4
language_model.model.layers.16 4
language_model.model.layers.17 4
language_model.model.layers.18 6
language_model.model.layers.19 6
language_model.model.layers.20 6
language_model.model.layers.21 6
language_model.model.layers.22 6
language_model.model.layers.23 6
vision_model.encoder.layers.0 0
vision_model.encoder.layers.1 0
vision_model.encoder.layers.2 0
vision_model.encoder.layers.3 0
vision_model.encoder.layers.4 0
vision_model.encoder.layers.5 0
vision_model.encoder.layers.6 0
vision_model.encoder.layers.7 0
vision_model.encoder.layers.8 0
vision_model.encoder.layers.9 0
vision_model.encoder.layers.10 0
vision_model.encoder.layers.11 0
vision_model.encoder.layers.12 0
vision_model.encoder.layers.13 0
vision_model.encoder.layers.14 0
vision_model.encoder.layers.15 0
vision_model.encoder.layers.16 0
vision_model.encoder.layers.17 0
vision_model.encoder.layers.18 0
vision_model.encoder.layers.19 0
vision_model.encoder.layers.20 0
vision_model.encoder.layers.21 0
vision_model.encoder.layers.22 0
vision_model.encoder.layers.23 0
vision_model.embeddings 0
mlp1 0
language_model.model.tok_embeddings 2
language_model.model.norm 6
language_model.output 6
language_model.model.embed_tokens 2
language_model.lm_head 6
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Rank [0] Begin to eval model /mnt/petrelfs/wangweiyun/workspace_zyc/VLM-Dev/pretrained/InternVL2-1B on task reasoning-image-val, devices: {device(type='cuda', index=0), device(type='cuda', index=2), device(type='cuda', index=4), device(type='cuda', index=6)}
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Rank [1] Begin to eval model /mnt/petrelfs/wangweiyun/workspace_zyc/VLM-Dev/pretrained/InternVL2-1B on task reasoning-image-val, devices: {device(type='cuda', index=1), device(type='cuda', index=3), device(type='cuda', index=5), device(type='cuda', index=7)}
Rank 1 len(skip_idx)=0
Rank 0 len(skip_idx)=0

Processing InternVL2-1B_reasoning-image-val.jsonl:   0%|          | 0/260 [00:00<?, ?it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:04] [Rank 0] totoal_tokens=837, outputs='A'
[2024-08-06 12:34:04] [Rank 1] totoal_tokens=837, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   0%|          | 1/260 [00:00<04:05,  1.06it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:04] [Rank 0] totoal_tokens=837, outputs='A'
[2024-08-06 12:34:04] [Rank 1] totoal_tokens=837, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   1%|          | 2/260 [00:01<02:01,  2.12it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:04] [Rank 0] totoal_tokens=837, outputs='A'
[2024-08-06 12:34:04] [Rank 1] totoal_tokens=862, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   1%|          | 3/260 [00:01<01:21,  3.14it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:04] [Rank 0] totoal_tokens=837, outputs='A'
[2024-08-06 12:34:04] [Rank 1] totoal_tokens=968, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   2%|▏         | 4/260 [00:01<01:02,  4.07it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:04] [Rank 0] totoal_tokens=925, outputs='A'
[2024-08-06 12:34:04] [Rank 1] totoal_tokens=975, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   2%|▏         | 5/260 [00:01<00:52,  4.84it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:04] [Rank 0] totoal_tokens=941, outputs='A'
[2024-08-06 12:34:04] [Rank 1] totoal_tokens=1000, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   2%|▏         | 6/260 [00:01<00:46,  5.44it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 0] totoal_tokens=950, outputs='A'
[2024-08-06 12:34:05] [Rank 1] totoal_tokens=1028, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   3%|▎         | 7/260 [00:01<00:42,  5.96it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 0] totoal_tokens=1025, outputs='A'
[2024-08-06 12:34:05] [Rank 1] totoal_tokens=1060, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   3%|▎         | 8/260 [00:01<00:39,  6.31it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 0] totoal_tokens=1082, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   3%|▎         | 9/260 [00:02<00:38,  6.57it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 1] totoal_tokens=1060, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 0] totoal_tokens=1120, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   4%|▍         | 10/260 [00:02<00:38,  6.53it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 1] totoal_tokens=1060, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 0] totoal_tokens=1120, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   4%|▍         | 11/260 [00:02<00:37,  6.55it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 1] totoal_tokens=1145, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 0] totoal_tokens=1120, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   5%|▍         | 12/260 [00:02<00:37,  6.57it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 1] totoal_tokens=1208, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:05] [Rank 0] totoal_tokens=1145, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   5%|▌         | 13/260 [00:02<00:37,  6.59it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 1] totoal_tokens=1208, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 0] totoal_tokens=1208, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   5%|▌         | 14/260 [00:02<00:38,  6.47it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 1] totoal_tokens=1208, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 0] totoal_tokens=1219, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   6%|▌         | 15/260 [00:02<00:37,  6.57it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 1] totoal_tokens=1219, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 0] totoal_tokens=1231, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   6%|▌         | 16/260 [00:03<00:35,  6.79it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 1] totoal_tokens=1277, outputs='A'
[2024-08-06 12:34:06] [Rank 0] totoal_tokens=1231, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:   7%|▋         | 17/260 [00:03<00:34,  6.97it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 1] totoal_tokens=1338, outputs='A'
[2024-08-06 12:34:06] [Rank 0] totoal_tokens=1233, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   7%|▋         | 18/260 [00:03<00:35,  6.76it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 0] totoal_tokens=1283, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   7%|▋         | 19/260 [00:03<00:36,  6.63it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:06] [Rank 0] totoal_tokens=1326, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   8%|▊         | 20/260 [00:03<00:36,  6.63it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:07] [Rank 0] totoal_tokens=1356, outputs='A'
[2024-08-06 12:34:07] [Rank 1] totoal_tokens=1354, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   8%|▊         | 21/260 [00:03<00:36,  6.62it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:07] [Rank 0] totoal_tokens=1366, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   8%|▊         | 22/260 [00:04<00:36,  6.61it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:07] [Rank 0] totoal_tokens=1367, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   9%|▉         | 23/260 [00:04<00:37,  6.27it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:07] [Rank 0] totoal_tokens=1403, outputs='A'
[2024-08-06 12:34:07] [Rank 1] totoal_tokens=1354, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:   9%|▉         | 24/260 [00:04<00:37,  6.27it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:07] [Rank 0] totoal_tokens=1503, outputs='A'
[2024-08-06 12:34:07] [Rank 1] totoal_tokens=1361, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  10%|▉         | 25/260 [00:04<00:36,  6.37it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:07] [Rank 0] totoal_tokens=1553, outputs='A'
[2024-08-06 12:34:07] [Rank 1] totoal_tokens=1367, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  10%|█         | 26/260 [00:04<00:37,  6.26it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:08] [Rank 0] totoal_tokens=1696, outputs='A'
[2024-08-06 12:34:08] [Rank 1] totoal_tokens=1381, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  10%|█         | 27/260 [00:04<00:37,  6.15it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:08] [Rank 0] totoal_tokens=1783, outputs='A'
[2024-08-06 12:34:08] [Rank 1] totoal_tokens=1431, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  11%|█         | 28/260 [00:05<00:40,  5.67it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:08] [Rank 1] totoal_tokens=1450, outputs='A'
[2024-08-06 12:34:08] [Rank 0] totoal_tokens=1948, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  11%|█         | 29/260 [00:05<00:43,  5.32it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:08] [Rank 1] totoal_tokens=1503, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:08] [Rank 0] totoal_tokens=2267, outputs='A'
[2024-08-06 12:34:08] [Rank 1] totoal_tokens=1564, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  12%|█▏        | 30/260 [00:05<00:45,  5.10it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:08] [Rank 0] totoal_tokens=2292, outputs='A'
[2024-08-06 12:34:08] [Rank 1] totoal_tokens=1750, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  12%|█▏        | 31/260 [00:05<00:46,  4.93it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:09] [Rank 0] totoal_tokens=2367, outputs='A'
[2024-08-06 12:34:09] [Rank 1] totoal_tokens=2070, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  12%|█▏        | 32/260 [00:05<00:46,  4.87it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:09] [Rank 0] totoal_tokens=2457, outputs='A'
[2024-08-06 12:34:09] [Rank 1] totoal_tokens=2221, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  13%|█▎        | 33/260 [00:06<00:48,  4.70it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:09] [Rank 0] totoal_tokens=2518, outputs='A'
[2024-08-06 12:34:09] [Rank 1] totoal_tokens=2221, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  13%|█▎        | 34/260 [00:06<00:50,  4.52it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:09] [Rank 1] totoal_tokens=2239, outputs='A'
[2024-08-06 12:34:09] [Rank 0] totoal_tokens=2518, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  13%|█▎        | 35/260 [00:06<00:51,  4.41it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:10] [Rank 1] totoal_tokens=2267, outputs='A'
[2024-08-06 12:34:10] [Rank 0] totoal_tokens=2518, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  14%|█▍        | 36/260 [00:06<00:51,  4.34it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:10] [Rank 1] totoal_tokens=2278, outputs='A'
[2024-08-06 12:34:10] [Rank 0] totoal_tokens=2520, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  14%|█▍        | 37/260 [00:07<00:51,  4.35it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:10] [Rank 1] totoal_tokens=2402, outputs='A'
[2024-08-06 12:34:10] [Rank 0] totoal_tokens=2556, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  15%|█▍        | 38/260 [00:07<00:51,  4.31it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:10] [Rank 0] totoal_tokens=2746, outputs='A'
[2024-08-06 12:34:10] [Rank 1] totoal_tokens=2425, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  15%|█▌        | 39/260 [00:07<00:44,  4.91it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:10] [Rank 0] totoal_tokens=2746, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  15%|█▌        | 40/260 [00:07<00:39,  5.52it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:10] [Rank 0] totoal_tokens=2746, outputs='A'
[2024-08-06 12:34:10] [Rank 1] totoal_tokens=2504, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  16%|█▌        | 41/260 [00:07<00:36,  6.04it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 0] totoal_tokens=2803, outputs='A'
[2024-08-06 12:34:11] [Rank 1] totoal_tokens=2615, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  16%|█▌        | 42/260 [00:07<00:34,  6.40it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 0] totoal_tokens=2838, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  17%|█▋        | 43/260 [00:07<00:32,  6.69it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 1] totoal_tokens=2618, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 1] totoal_tokens=2728, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 0] totoal_tokens=2842, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  17%|█▋        | 44/260 [00:08<00:41,  5.16it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 1] totoal_tokens=2728, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 0] totoal_tokens=2898, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  17%|█▋        | 45/260 [00:08<00:37,  5.67it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 1] totoal_tokens=2746, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 1] totoal_tokens=2746, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:11] [Rank 0] totoal_tokens=2972, outputs='A'
[2024-08-06 12:34:12] [Rank 1] totoal_tokens=2838, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  18%|█▊        | 46/260 [00:08<00:45,  4.67it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:12] [Rank 1] totoal_tokens=2838, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:12] [Rank 0] totoal_tokens=2996, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  18%|█▊        | 47/260 [00:09<00:50,  4.19it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:12] [Rank 1] totoal_tokens=2905, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:12] [Rank 0] totoal_tokens=3028, outputs='A'
[2024-08-06 12:34:12] [Rank 1] totoal_tokens=2986, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  18%|█▊        | 48/260 [00:09<00:54,  3.90it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:12] [Rank 0] totoal_tokens=3055, outputs='A'
[2024-08-06 12:34:12] [Rank 1] totoal_tokens=2988, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  19%|█▉        | 49/260 [00:09<00:46,  4.59it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:12] [Rank 0] totoal_tokens=3055, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  19%|█▉        | 50/260 [00:09<00:40,  5.23it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:13] [Rank 1] totoal_tokens=2992, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:13] [Rank 0] totoal_tokens=3118, outputs='A'
[2024-08-06 12:34:13] [Rank 1] totoal_tokens=3009, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  20%|█▉        | 51/260 [00:09<00:46,  4.47it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:13] [Rank 1] totoal_tokens=3055, outputs='A'
[2024-08-06 12:34:13] [Rank 0] totoal_tokens=3184, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  20%|██        | 52/260 [00:10<00:43,  4.73it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:13] [Rank 1] totoal_tokens=3118, outputs='A'
[2024-08-06 12:34:13] [Rank 0] totoal_tokens=3375, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  20%|██        | 53/260 [00:10<00:49,  4.17it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:13] [Rank 1] totoal_tokens=3520, outputs='B'
[2024-08-06 12:34:13] [Rank 0] totoal_tokens=3569, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  21%|██        | 54/260 [00:10<00:55,  3.74it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:14] [Rank 1] totoal_tokens=3521, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:14] [Rank 0] totoal_tokens=3733, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  21%|██        | 55/260 [00:11<01:00,  3.39it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:14] [Rank 1] totoal_tokens=3846, outputs='A'
[2024-08-06 12:34:14] [Rank 0] totoal_tokens=4217, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  22%|██▏       | 56/260 [00:11<00:54,  3.72it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:14] [Rank 0] totoal_tokens=4274, outputs='A'
[2024-08-06 12:34:14] [Rank 1] totoal_tokens=3976, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  22%|██▏       | 57/260 [00:11<00:50,  4.04it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:14] [Rank 1] totoal_tokens=3999, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:15] [Rank 0] totoal_tokens=4355, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  22%|██▏       | 58/260 [00:11<01:04,  3.14it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:15] [Rank 1] totoal_tokens=4195, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:15] [Rank 0] totoal_tokens=4471, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  23%|██▎       | 59/260 [00:12<01:08,  2.92it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:15] [Rank 1] totoal_tokens=4223, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:16] [Rank 0] totoal_tokens=4598, outputs='A'
[2024-08-06 12:34:16] [Rank 1] totoal_tokens=4276, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  23%|██▎       | 60/260 [00:12<01:13,  2.74it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:16] [Rank 1] totoal_tokens=4283, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:16] [Rank 0] totoal_tokens=4659, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  23%|██▎       | 61/260 [00:13<01:44,  1.91it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:17] [Rank 1] totoal_tokens=4471, outputs='A'
[2024-08-06 12:34:17] [Rank 0] totoal_tokens=4782, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  24%|██▍       | 62/260 [00:13<01:29,  2.20it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:17] [Rank 1] totoal_tokens=4625, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:17] [Rank 1] totoal_tokens=4635, outputs='A'
[2024-08-06 12:34:17] [Rank 0] totoal_tokens=4829, outputs='B'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  24%|██▍       | 63/260 [00:14<01:28,  2.22it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:17] [Rank 0] totoal_tokens=5203, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  25%|██▍       | 64/260 [00:14<01:17,  2.53it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:18] [Rank 1] totoal_tokens=4827, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:18] [Rank 0] totoal_tokens=5331, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  25%|██▌       | 65/260 [00:15<01:23,  2.33it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:18] [Rank 1] totoal_tokens=5209, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:18] [Rank 0] totoal_tokens=5377, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  25%|██▌       | 66/260 [00:15<01:14,  2.60it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:18] [Rank 1] totoal_tokens=5270, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:19] [Rank 1] totoal_tokens=5377, outputs='A'
[2024-08-06 12:34:19] [Rank 0] totoal_tokens=5389, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  26%|██▌       | 67/260 [00:15<01:18,  2.47it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:19] [Rank 0] totoal_tokens=5428, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  26%|██▌       | 68/260 [00:16<01:13,  2.63it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:19] [Rank 1] totoal_tokens=5397, outputs='B'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:19] [Rank 0] totoal_tokens=5436, outputs='A'
[2024-08-06 12:34:20] [Rank 1] totoal_tokens=5397, outputs='B'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  27%|██▋       | 69/260 [00:16<01:09,  2.75it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:20] [Rank 1] totoal_tokens=5398, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:20] [Rank 0] totoal_tokens=5450, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  27%|██▋       | 70/260 [00:17<01:39,  1.90it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:20] [Rank 1] totoal_tokens=5425, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:21] [Rank 0] totoal_tokens=5497, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  27%|██▋       | 71/260 [00:17<01:37,  1.95it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:21] [Rank 1] totoal_tokens=5451, outputs='A'
[2024-08-06 12:34:21] [Rank 0] totoal_tokens=5497, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  28%|██▊       | 72/260 [00:18<01:26,  2.17it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:21] [Rank 0] totoal_tokens=5545, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  28%|██▊       | 73/260 [00:18<01:17,  2.42it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:21] [Rank 1] totoal_tokens=5735, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:22] [Rank 0] totoal_tokens=5699, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  28%|██▊       | 74/260 [00:18<01:13,  2.53it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:22] [Rank 1] totoal_tokens=5901, outputs='A'
[2024-08-06 12:34:22] [Rank 0] totoal_tokens=5801, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  29%|██▉       | 75/260 [00:19<01:09,  2.66it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:23] [Rank 1] totoal_tokens=5919, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:23] [Rank 0] totoal_tokens=6282, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  29%|██▉       | 76/260 [00:19<01:27,  2.11it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:23] [Rank 1] totoal_tokens=6427, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:23] [Rank 0] totoal_tokens=6399, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  30%|██▉       | 77/260 [00:20<01:43,  1.77it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:24] [Rank 0] totoal_tokens=6617, outputs='A'
[2024-08-06 12:34:24] [Rank 1] totoal_tokens=6429, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  30%|███       | 78/260 [00:21<01:36,  1.89it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:24] [Rank 0] totoal_tokens=7033, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  30%|███       | 79/260 [00:21<01:33,  1.94it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:25] [Rank 1] totoal_tokens=6630, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:25] [Rank 0] totoal_tokens=7348, outputs='A'
[2024-08-06 12:34:25] [Rank 1] totoal_tokens=6737, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  31%|███       | 80/260 [00:22<01:41,  1.77it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:26] [Rank 1] totoal_tokens=6768, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:26] [Rank 0] totoal_tokens=7524, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  31%|███       | 81/260 [00:23<01:55,  1.55it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:26] [Rank 1] totoal_tokens=6945, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:27] [Rank 0] totoal_tokens=7713, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  32%|███▏      | 82/260 [00:23<01:56,  1.53it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:27] [Rank 1] totoal_tokens=7109, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:27] [Rank 1] totoal_tokens=7121, outputs='A'
[2024-08-06 12:34:27] [Rank 0] totoal_tokens=7813, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  32%|███▏      | 83/260 [00:24<02:02,  1.44it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:28] [Rank 1] totoal_tokens=7367, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:28] [Rank 0] totoal_tokens=8031, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  32%|███▏      | 84/260 [00:25<02:15,  1.30it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:29] [Rank 1] totoal_tokens=7468, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:29] [Rank 0] totoal_tokens=8132, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  33%|███▎      | 85/260 [00:26<02:15,  1.29it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:30] [Rank 1] totoal_tokens=7557, outputs='A'
[2024-08-06 12:34:30] [Rank 0] totoal_tokens=8192, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  33%|███▎      | 86/260 [00:26<02:06,  1.38it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:31] [Rank 0] totoal_tokens=8362, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  33%|███▎      | 87/260 [00:27<02:08,  1.35it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:31] [Rank 1] totoal_tokens=7818, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:31] [Rank 0] totoal_tokens=8417, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  34%|███▍      | 88/260 [00:28<01:57,  1.46it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:32] [Rank 1] totoal_tokens=8141, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:32] [Rank 0] totoal_tokens=8453, outputs='A'
[2024-08-06 12:34:32] [Rank 1] totoal_tokens=8581, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  34%|███▍      | 89/260 [00:29<02:12,  1.29it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:33] [Rank 0] totoal_tokens=8786, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  35%|███▍      | 90/260 [00:30<02:15,  1.25it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:33] [Rank 1] totoal_tokens=8786, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:34] [Rank 0] totoal_tokens=8827, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  35%|███▌      | 91/260 [00:30<02:06,  1.34it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:34] [Rank 1] totoal_tokens=8898, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:35] [Rank 0] totoal_tokens=8904, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  35%|███▌      | 92/260 [00:31<02:17,  1.23it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:35] [Rank 1] totoal_tokens=9029, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:36] [Rank 0] totoal_tokens=8933, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  36%|███▌      | 93/260 [00:32<02:25,  1.15it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:36] [Rank 1] totoal_tokens=9193, outputs='A'
[2024-08-06 12:34:36] [Rank 0] totoal_tokens=9148, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.

Processing InternVL2-1B_reasoning-image-val.jsonl:  36%|███▌      | 94/260 [00:33<02:19,  1.19it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:37] [Rank 1] totoal_tokens=9309, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:38] [Rank 0] totoal_tokens=9261, outputs='A'

Processing InternVL2-1B_reasoning-image-val.jsonl:  37%|███▋      | 95/260 [00:34<02:43,  1.01it/s]Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:38] [Rank 1] totoal_tokens=9453, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:39] [Rank 1] totoal_tokens=9532, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:40] [Rank 1] totoal_tokens=9579, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:41] [Rank 1] totoal_tokens=9837, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:42] [Rank 1] totoal_tokens=9837, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:43] [Rank 1] totoal_tokens=9994, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:45] [Rank 1] totoal_tokens=10459, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:46] [Rank 1] totoal_tokens=10519, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:48] [Rank 1] totoal_tokens=10545, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:50] [Rank 1] totoal_tokens=10716, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:51] [Rank 1] totoal_tokens=10861, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:53] [Rank 1] totoal_tokens=11056, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:54] [Rank 1] totoal_tokens=11069, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:56] [Rank 1] totoal_tokens=11226, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:34:57] [Rank 1] totoal_tokens=11394, outputs='B'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:00] [Rank 1] totoal_tokens=11440, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:01] [Rank 1] totoal_tokens=11808, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:02] [Rank 1] totoal_tokens=11813, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:03] [Rank 1] totoal_tokens=11944, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:05] [Rank 1] totoal_tokens=12144, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:07] [Rank 1] totoal_tokens=12678, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:09] [Rank 1] totoal_tokens=12841, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:10] [Rank 1] totoal_tokens=12999, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:12] [Rank 1] totoal_tokens=13033, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:13] [Rank 1] totoal_tokens=13114, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:15] [Rank 1] totoal_tokens=13135, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:17] [Rank 1] totoal_tokens=13135, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:19] [Rank 1] totoal_tokens=13307, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:22] [Rank 1] totoal_tokens=13704, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:23] [Rank 1] totoal_tokens=13980, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:25] [Rank 1] totoal_tokens=14089, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:28] [Rank 1] totoal_tokens=14644, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:30] [Rank 1] totoal_tokens=14721, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:32] [Rank 1] totoal_tokens=15173, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:34] [Rank 1] totoal_tokens=15289, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:36] [Rank 1] totoal_tokens=15340, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:38] [Rank 1] totoal_tokens=15494, outputs='B'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:40] [Rank 1] totoal_tokens=15590, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:42] [Rank 1] totoal_tokens=15755, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:44] [Rank 1] totoal_tokens=16511, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:51] [Rank 1] totoal_tokens=16781, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:55] [Rank 1] totoal_tokens=16982, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:35:58] [Rank 1] totoal_tokens=17254, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:01] [Rank 1] totoal_tokens=17326, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:08] [Rank 1] totoal_tokens=17447, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:11] [Rank 1] totoal_tokens=17464, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:15] [Rank 1] totoal_tokens=17672, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:23] [Rank 1] totoal_tokens=17803, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:25] [Rank 1] totoal_tokens=17896, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:28] [Rank 1] totoal_tokens=18027, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:31] [Rank 1] totoal_tokens=18231, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:35] [Rank 1] totoal_tokens=18672, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:39] [Rank 1] totoal_tokens=18850, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:42] [Rank 1] totoal_tokens=20072, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:46] [Rank 1] totoal_tokens=20218, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:36:50] [Rank 1] totoal_tokens=20419, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:00] [Rank 1] totoal_tokens=20509, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:03] [Rank 1] totoal_tokens=20558, outputs='B'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:07] [Rank 1] totoal_tokens=20562, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:11] [Rank 1] totoal_tokens=21431, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:19] [Rank 1] totoal_tokens=21537, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:23] [Rank 1] totoal_tokens=21628, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:28] [Rank 1] totoal_tokens=22701, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:39] [Rank 1] totoal_tokens=23289, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:48] [Rank 1] totoal_tokens=23416, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:53] [Rank 1] totoal_tokens=23692, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:37:59] [Rank 1] totoal_tokens=24166, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:38:10] [Rank 1] totoal_tokens=24170, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:38:16] [Rank 1] totoal_tokens=24203, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:38:26] [Rank 1] totoal_tokens=24401, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:38:39] [Rank 1] totoal_tokens=24800, outputs='B'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:38:44] [Rank 1] totoal_tokens=25643, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:38:50] [Rank 1] totoal_tokens=26154, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:39:05] [Rank 1] totoal_tokens=27430, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:39:13] [Rank 1] totoal_tokens=27528, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:39:22] [Rank 1] totoal_tokens=28155, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=28363, error: CUDA out of memory. Tried to allocate 27.94 GiB. GPU 3 has a total capacty of 79.33 GiB of which 22.45 GiB is free. Including non-PyTorch memory, this process has 56.86 GiB memory in use. Of the allocated memory 43.70 GiB is allocated by PyTorch, and 12.66 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:30] [Rank 1] totoal_tokens=28363, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=28420, error: CUDA out of memory. Tried to allocate 29.49 GiB. GPU 3 has a total capacty of 79.33 GiB of which 20.91 GiB is free. Including non-PyTorch memory, this process has 58.41 GiB memory in use. Of the allocated memory 46.08 GiB is allocated by PyTorch, and 11.83 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:31] [Rank 1] totoal_tokens=28420, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=28440, error: CUDA out of memory. Tried to allocate 27.01 GiB. GPU 3 has a total capacty of 79.33 GiB of which 23.39 GiB is free. Including non-PyTorch memory, this process has 55.93 GiB memory in use. Of the allocated memory 42.27 GiB is allocated by PyTorch, and 13.17 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:33] [Rank 1] totoal_tokens=28440, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=28455, error: CUDA out of memory. Tried to allocate 27.10 GiB. GPU 3 has a total capacty of 79.33 GiB of which 23.30 GiB is free. Including non-PyTorch memory, this process has 56.01 GiB memory in use. Of the allocated memory 42.40 GiB is allocated by PyTorch, and 13.12 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:35] [Rank 1] totoal_tokens=28455, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[2024-08-06 12:39:43] [Rank 1] totoal_tokens=28464, outputs='A'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=29525, error: CUDA out of memory. Tried to allocate 28.66 GiB. GPU 3 has a total capacty of 79.33 GiB of which 23.13 GiB is free. Including non-PyTorch memory, this process has 56.18 GiB memory in use. Of the allocated memory 44.80 GiB is allocated by PyTorch, and 10.89 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:46] [Rank 1] totoal_tokens=29525, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=29830, error: CUDA out of memory. Tried to allocate 28.50 GiB. GPU 3 has a total capacty of 79.33 GiB of which 23.29 GiB is free. Including non-PyTorch memory, this process has 56.03 GiB memory in use. Of the allocated memory 44.56 GiB is allocated by PyTorch, and 10.97 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:47] [Rank 1] totoal_tokens=29830, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=30193, error: CUDA out of memory. Tried to allocate 26.36 GiB. GPU 3 has a total capacty of 79.33 GiB of which 25.43 GiB is free. Including non-PyTorch memory, this process has 53.88 GiB memory in use. Of the allocated memory 41.26 GiB is allocated by PyTorch, and 12.12 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:49] [Rank 1] totoal_tokens=30193, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=31240, error: CUDA out of memory. Tried to allocate 28.36 GiB. GPU 3 has a total capacty of 79.33 GiB of which 23.43 GiB is free. Including non-PyTorch memory, this process has 55.89 GiB memory in use. Of the allocated memory 44.35 GiB is allocated by PyTorch, and 11.04 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:51] [Rank 1] totoal_tokens=31240, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=31411, error: CUDA out of memory. Tried to allocate 31.79 GiB. GPU 3 has a total capacty of 79.33 GiB of which 20.00 GiB is free. Including non-PyTorch memory, this process has 59.31 GiB memory in use. Of the allocated memory 49.62 GiB is allocated by PyTorch, and 9.20 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:53] [Rank 1] totoal_tokens=31411, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=31413, error: CUDA out of memory. Tried to allocate 28.03 GiB. GPU 3 has a total capacty of 79.33 GiB of which 23.76 GiB is free. Including non-PyTorch memory, this process has 55.55 GiB memory in use. Of the allocated memory 43.83 GiB is allocated by PyTorch, and 11.23 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:55] [Rank 1] totoal_tokens=31413, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=32140, error: CUDA out of memory. Tried to allocate 30.46 GiB. GPU 3 has a total capacty of 79.33 GiB of which 21.33 GiB is free. Including non-PyTorch memory, this process has 57.98 GiB memory in use. Of the allocated memory 47.58 GiB is allocated by PyTorch, and 9.91 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:39:57] [Rank 1] totoal_tokens=32140, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=32464, error: CUDA out of memory. Tried to allocate 31.60 GiB. GPU 3 has a total capacty of 79.33 GiB of which 20.19 GiB is free. Including non-PyTorch memory, this process has 59.12 GiB memory in use. Of the allocated memory 49.33 GiB is allocated by PyTorch, and 9.30 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:00] [Rank 1] totoal_tokens=32464, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=33130, error: CUDA out of memory. Tried to allocate 31.20 GiB. GPU 3 has a total capacty of 79.33 GiB of which 20.59 GiB is free. Including non-PyTorch memory, this process has 58.72 GiB memory in use. Of the allocated memory 48.71 GiB is allocated by PyTorch, and 9.52 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:02] [Rank 1] totoal_tokens=33130, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=33268, error: CUDA out of memory. Tried to allocate 32.89 GiB. GPU 3 has a total capacty of 79.33 GiB of which 18.90 GiB is free. Including non-PyTorch memory, this process has 60.42 GiB memory in use. Of the allocated memory 51.32 GiB is allocated by PyTorch, and 8.60 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:04] [Rank 1] totoal_tokens=33268, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=33355, error: CUDA out of memory. Tried to allocate 36.55 GiB. GPU 3 has a total capacty of 79.33 GiB of which 15.24 GiB is free. Including non-PyTorch memory, this process has 64.07 GiB memory in use. Of the allocated memory 56.95 GiB is allocated by PyTorch, and 6.63 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:06] [Rank 1] totoal_tokens=33355, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=33614, error: CUDA out of memory. Tried to allocate 38.56 GiB. GPU 3 has a total capacty of 79.33 GiB of which 13.23 GiB is free. Including non-PyTorch memory, this process has 66.09 GiB memory in use. Of the allocated memory 60.05 GiB is allocated by PyTorch, and 5.55 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:09] [Rank 1] totoal_tokens=33614, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=34700, error: CUDA out of memory. Tried to allocate 33.87 GiB. GPU 3 has a total capacty of 79.33 GiB of which 17.92 GiB is free. Including non-PyTorch memory, this process has 61.39 GiB memory in use. Of the allocated memory 52.83 GiB is allocated by PyTorch, and 8.07 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:11] [Rank 1] totoal_tokens=34700, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=35419, error: CUDA out of memory. Tried to allocate 54.28 GiB. GPU 3 has a total capacty of 79.33 GiB of which 48.36 GiB is free. Including non-PyTorch memory, this process has 30.95 GiB memory in use. Of the allocated memory 29.95 GiB is allocated by PyTorch, and 520.63 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:13] [Rank 1] totoal_tokens=35419, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=37167, error: CUDA out of memory. Tried to allocate 40.67 GiB. GPU 3 has a total capacty of 79.33 GiB of which 7.69 GiB is free. Including non-PyTorch memory, this process has 71.62 GiB memory in use. Of the allocated memory 63.29 GiB is allocated by PyTorch, and 7.84 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:15] [Rank 1] totoal_tokens=37167, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=38390, error: CUDA out of memory. Tried to allocate 42.88 GiB. GPU 3 has a total capacty of 79.33 GiB of which 5.48 GiB is free. Including non-PyTorch memory, this process has 73.83 GiB memory in use. Of the allocated memory 66.69 GiB is allocated by PyTorch, and 6.65 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:18] [Rank 1] totoal_tokens=38390, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=38600, error: CUDA out of memory. Tried to allocate 54.25 GiB. GPU 3 has a total capacty of 79.33 GiB of which 48.36 GiB is free. Including non-PyTorch memory, this process has 30.95 GiB memory in use. Of the allocated memory 29.94 GiB is allocated by PyTorch, and 531.74 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:20] [Rank 1] totoal_tokens=38600, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=39378, error: CUDA out of memory. Tried to allocate 53.59 GiB. GPU 3 has a total capacty of 79.33 GiB of which 48.36 GiB is free. Including non-PyTorch memory, this process has 30.95 GiB memory in use. Of the allocated memory 29.58 GiB is allocated by PyTorch, and 899.58 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:22] [Rank 1] totoal_tokens=39378, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=39538, error: CUDA out of memory. Tried to allocate 57.58 GiB. GPU 3 has a total capacty of 79.33 GiB of which 46.77 GiB is free. Including non-PyTorch memory, this process has 32.55 GiB memory in use. Of the allocated memory 31.72 GiB is allocated by PyTorch, and 334.90 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:24] [Rank 1] totoal_tokens=39538, outputs='None'
Setting `pad_token_id` to `eos_token_id`:151645 for open-end generation.
[Rank 1] OutOfMemoryError occurs! totoal_tokens=39641, error: CUDA out of memory. Tried to allocate 54.41 GiB. GPU 3 has a total capacty of 79.33 GiB of which 46.77 GiB is free. Including non-PyTorch memory, this process has 32.55 GiB memory in use. Of the allocated memory 30.02 GiB is allocated by PyTorch, and 2.04 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2024-08-06 12:40:26] [Rank 1] totoal_tokens=39641, outputs='None'
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[Rank 1] early stops because of successive failures. oom_cnt=20
[2024-08-06 12:40:26] Rank 1 Finish
srun: Job step aborted: Waiting up to 2 seconds for job step to finish.
slurmstepd: error: *** JOB 3756917 ON HOST-10-140-60-138 CANCELLED AT 2024-08-06T13:43:53 ***
srun: Easily find out why your job was killed by following the link below:
	https://docs.phoenix.sensetime.com/FAQ/SlurmFAQ/Find-out-why-my-job-was-killed/
srun: got SIGCONT
srun: forcing job termination
slurmstepd: error: *** STEP 3756917.0 ON HOST-10-140-60-138 CANCELLED AT 2024-08-06T13:43:53 ***