how to run the model with vllm?
#2
by
sakura1225
- opened
run command: vllm serve AI4Chem/ChemLLM-7B-Chat-1_5-DPO -p 8082 --trust-remote-code
got error:
Traceback (most recent call last):
File "/home/anaconda3/envs/vllm/bin/vllm", line 8, in <module>
sys.exit(main())
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/scripts.py", line 156, in main
args.dispatch_function(args)
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/scripts.py", line 37, in serve
asyncio.run(run_server(args))
File "/home/anaconda3/envs/vllm/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/home/anaconda3/envs/vllm/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 443, in run_server
async with build_async_engine_client(args) as async_engine_client:
File "/home/anaconda3/envs/vllm/lib/python3.10/contextlib.py", line 199, in __aenter__
return await anext(self.gen)
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 117, in build_async_engine_client
if (model_is_embedding(args.model, args.trust_remote_code,
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 71, in model_is_embedding
return ModelConfig(model=model_name,
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/config.py", line 214, in __init__
self.max_model_len = _get_and_verify_max_len(
File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/config.py", line 1650, in _get_and_verify_max_len
assert "factor" in rope_scaling
AssertionError
package vsersion:
vllm 0.5.5
torch 2.4.0