You are using a model of type mixtral_aqlm to instantiate a model of type mixtral. This is not supported for all configurations of models and can yield errors.
#3
by
Tejasram
- opened
When I try to run this model using the huggingface transformers library, I get this warning. Is it safe to ignore?
Pleas use this with AutoModelForCausalLM:
from transformers import AutoTokenizer, AutoModelForCausalLM
quantized_model = AutoModelForCausalLM.from_pretrained(
"BlackSamorez/Mixtral-8x7b-AQLM-2Bit-1x16-hf",
trust_remote_code=True, torch_dtype="auto"
).cuda()
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B-v0.1")