logs/model_worker_5bf53a.log

2024-07-10 17:41:28 | INFO | model_worker | args: Namespace(host='0.0.0.0', port=40006, worker_address='http://10.140.60.25:40006', controller_address='http://10.140.60.209:10075', model_path='share_internvl/InternVL2-40B/', model_name=None, device='cuda', limit_model_concurrency=5, stream_interval=1, load_8bit=True)
2024-07-10 17:41:28 | INFO | model_worker | Loading the model InternVL2-40B on worker 5bf53a ...
2024-07-10 17:41:28 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-10 17:41:28 | WARNING | transformers.tokenization_utils_base | Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2024-07-10 17:41:29 | ERROR | stderr | /mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-10 17:41:29 | ERROR | stderr |   warnings.warn(
2024-07-10 17:41:36 | ERROR | stderr | 
Loading checkpoint shards:   0%|          | 0/17 [00:00<?, ?it/s]
2024-07-10 17:41:38 | ERROR | stderr | 
Loading checkpoint shards:   6%|▌         | 1/17 [00:02<00:38,  2.39s/it]
2024-07-10 17:41:40 | ERROR | stderr | 
Loading checkpoint shards:  12%|█▏        | 2/17 [00:04<00:34,  2.32s/it]
2024-07-10 17:41:42 | ERROR | stderr | 
Loading checkpoint shards:  18%|█▊        | 3/17 [00:06<00:31,  2.24s/it]
2024-07-10 17:41:44 | ERROR | stderr | 
Loading checkpoint shards:  24%|██▎       | 4/17 [00:08<00:28,  2.18s/it]
2024-07-10 17:41:47 | ERROR | stderr | 
Loading checkpoint shards:  29%|██▉       | 5/17 [00:11<00:26,  2.19s/it]
2024-07-10 17:41:49 | ERROR | stderr | 
Loading checkpoint shards:  35%|███▌      | 6/17 [00:13<00:23,  2.16s/it]
2024-07-10 17:41:51 | ERROR | stderr | 
Loading checkpoint shards:  41%|████      | 7/17 [00:15<00:21,  2.14s/it]
2024-07-10 17:41:53 | ERROR | stderr | 
Loading checkpoint shards:  47%|████▋     | 8/17 [00:17<00:19,  2.15s/it]
2024-07-10 17:41:55 | ERROR | stderr | 
Loading checkpoint shards:  53%|█████▎    | 9/17 [00:19<00:17,  2.14s/it]
2024-07-10 17:41:57 | ERROR | stderr | 
Loading checkpoint shards:  59%|█████▉    | 10/17 [00:21<00:14,  2.14s/it]
2024-07-10 17:41:59 | ERROR | stderr | 
Loading checkpoint shards:  65%|██████▍   | 11/17 [00:23<00:12,  2.16s/it]
2024-07-10 17:42:02 | ERROR | stderr | 
Loading checkpoint shards:  71%|███████   | 12/17 [00:26<00:10,  2.13s/it]
2024-07-10 17:42:04 | ERROR | stderr | 
Loading checkpoint shards:  76%|███████▋  | 13/17 [00:28<00:08,  2.16s/it]
2024-07-10 17:42:06 | ERROR | stderr | 
Loading checkpoint shards:  82%|████████▏ | 14/17 [00:30<00:06,  2.20s/it]
2024-07-10 17:42:08 | ERROR | stderr | 
Loading checkpoint shards:  88%|████████▊ | 15/17 [00:32<00:04,  2.18s/it]
2024-07-10 17:42:10 | ERROR | stderr | 
Loading checkpoint shards:  94%|█████████▍| 16/17 [00:34<00:02,  2.20s/it]
2024-07-10 17:42:12 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 17/17 [00:36<00:00,  1.92s/it]
2024-07-10 17:42:12 | ERROR | stderr | 
Loading checkpoint shards: 100%|██████████| 17/17 [00:36<00:00,  2.13s/it]
2024-07-10 17:42:12 | ERROR | stderr | 
2024-07-10 17:42:12 | INFO | model_worker | Register to controller
2024-07-10 17:42:12 | ERROR | stderr | INFO:     Started server process [19648]
2024-07-10 17:42:12 | ERROR | stderr | INFO:     Waiting for application startup.
2024-07-10 17:42:12 | ERROR | stderr | INFO:     Application startup complete.
2024-07-10 17:42:12 | ERROR | stderr | INFO:     Uvicorn running on http://0.0.0.0:40006 (Press CTRL+C to quit)
2024-07-10 17:42:27 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:42 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:42:57 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:12 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:27 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:42 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:43:57 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:12 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:27 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:42 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:44:57 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:12 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:27 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:42 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:45:57 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:12 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:27 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:42 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:46:57 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:12 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:47:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:48:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:49:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:50:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:51:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:52:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:53:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:54:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:55:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: None. global_counter: 0
2024-07-10 17:56:33 | INFO | stdout | INFO:     10.140.60.209:47632 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:36 | INFO | stdout | INFO:     10.140.60.209:47654 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:37 | INFO | stdout | INFO:     10.140.60.209:47672 - "POST /worker_get_status HTTP/1.1" 200 OK
2024-07-10 17:56:38 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-10 17:56:38 | INFO | stdout | INFO:     10.140.60.209:47678 - "POST /worker_generate_stream HTTP/1.1" 200 OK
2024-07-10 17:56:38 | INFO | stdout | history: []
2024-07-10 17:56:38 | INFO | stdout | question: Image-1: <image>
2024-07-10 17:56:38 | INFO | stdout | Describe this image in detail.
2024-07-10 17:56:38 | INFO | stdout | pil_images: [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=1024x1024 at 0x7F2C1E90C400>]
2024-07-10 17:56:38 | INFO | model_worker | dynamic_image_size: True
2024-07-10 17:56:38 | INFO | model_worker | use_thumbnail: True
2024-07-10 17:56:38 | INFO | model_worker | Resize images to 448x448
2024-07-10 17:56:38 | INFO | model_worker | Split images to torch.Size([10, 3, 448, 448])
2024-07-10 17:56:39 | ERROR | stderr | /mnt/petrelfs/wangweiyun/miniconda3/envs/internvl-apex/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:397: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `None` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
2024-07-10 17:56:39 | ERROR | stderr |   warnings.warn(
2024-07-10 17:56:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-10 17:56:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-10 17:57:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-10 17:57:28 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-10 17:57:43 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-10 17:57:58 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-10 17:58:13 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=4, locked=False). global_counter: 1
2024-07-10 17:58:24 | INFO | model_worker | Send heart beat. Models: ['InternVL2-40B']. Semaphore: Semaphore(value=5, locked=False). global_counter: 1