ModelCloud

/

Qwen2.5-Coder-32B-Instruct-gptqmodel-4bit-vortex-v1

like 8

Follow

ModelCloud.AI 14

Is this a marlin kernel compatible?

Yes, but you need to use GPTQModel.load() and pass the parameter backend=BACKEND.MARLIN. It will use the Marlin kernel compatible with group_size 32.