Speculative decoding with Qwen 32B + Qwen 1.5B

Example:

llama-server \
    -m qwen2.5-coder-32b-instruct-q4_k_m.gguf \
    -md qwen2.5-coder-1.5b-instruct-q4_k_m.gguf \
    --ctx-size 65536

GGUF

Model size

1.78B params

Architecture

qwen2

4-bit

Inference Providers NEW

This model is not currently available via any of the supported Inference Providers.

The model cannot be deployed to the HF Inference API: The HF Inference API does not support text-generation models for gguf library.

Model tree for ggml-org/Qwen2.5-Coder-1.5B-32B-speculative-GGUF

Base model

Qwen/Qwen2.5-32B

Finetuned

Finetuned

Quantized

(93)

this model