tflsxyy/DeepSeek-V3-4bit-4layers

This is the first 4 layers of DeepSeek-V3 with GPTQ quantization style.

• Layer 4's all routed experts (256 experts) are quantized to 2-bit
• All other Linear layers are quantized to 4-bit (including MLA, dense layer ffn, and shared expert)

To load and run this model:

from transformers import AutoModelForCausalLM, AutoTokenizer
from gptqmodel import GPTQModel, QuantizeConfig, get_best_device

pretrained_model_id = "/root/dataDisk/DeepSeek-V3-bf16-4layers"
quantized_model_id = "/root/dataDisk/DeepSeek-V3-4bit-4layers"

tokenizer = AutoTokenizer.from_pretrained(pretrained_model_id, use_fast=True)
device = get_best_device()
model = GPTQModel.load(quantized_model_id, device=device, trust_remote_code=True)
print(tokenizer.decode(model.generate(**tokenizer("gptqmodel is", return_tensors="pt").to(model.device), max_new_tokens=10)[0]))