how to run the model with vllm?

#2
by sakura1225 - opened

run command: vllm serve AI4Chem/ChemLLM-7B-Chat-1_5-DPO -p 8082 --trust-remote-code
got error:

Traceback (most recent call last):
  File "/home/anaconda3/envs/vllm/bin/vllm", line 8, in <module>
    sys.exit(main())
  File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/scripts.py", line 156, in main
    args.dispatch_function(args)
  File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/scripts.py", line 37, in serve
    asyncio.run(run_server(args))
  File "/home/anaconda3/envs/vllm/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/anaconda3/envs/vllm/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 443, in run_server
    async with build_async_engine_client(args) as async_engine_client:
  File "/home/anaconda3/envs/vllm/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 117, in build_async_engine_client
    if (model_is_embedding(args.model, args.trust_remote_code,
  File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 71, in model_is_embedding
    return ModelConfig(model=model_name,
  File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/config.py", line 214, in __init__
    self.max_model_len = _get_and_verify_max_len(
  File "/home/anaconda3/envs/vllm/lib/python3.10/site-packages/vllm/config.py", line 1650, in _get_and_verify_max_len
    assert "factor" in rope_scaling
AssertionError

package vsersion:
vllm 0.5.5
torch 2.4.0

Sign up or log in to comment