bartowski/Llama-3.3-70B-Instruct-GGUF · Problem in running with vllm

27 days ago

Hi
Command vllm serve "bartowski/Llama-3.3-70B-Instruct-GGUF" returns following error. how to fix this?

raise ValueError(f"No supported config format found in {model}")
ValueError: No supported config format found in bartowski/Llama-3.3-70B-Instruct-GGUF

bartowski

Owner 27 days ago

i think you need to add a config.json file with:

{
    "model_type": "llama"
}

can you confirm if this works?

hliang123456789

3 days ago

Hi,

I'm having the same issue. I tried downloading the Llama-3.3-70B-Instruct-Q6_K_L file as well, and I combined them using llama-gguf-split --merge. The file is fine as I tried loading it with llama-cpp-python and it works. When I run

vllm serve Llama3-3.3-70B-Instruct-Q6_K_L.gguf --tokenizer=meta-llama/Llama-3.3-70B-Instruct --api-key=<KEY> --max-model-len=32000

I get

ValueError: No supported config format found in Llama3-3.3-70B-Instruct-Q6_K_L.gguf

I tried putting a config.json file in the same directory as the model with the elements you said above but it doesn't work. Please advise. Thanks!