bartowski

/

granite-3.0-8b-instruct-GGUF

like 11

Hi, I'm new to quantization. I'm trying to use the Q6_K GGUF file with the below code but I'm getting this error. Can you please help in resolving this issue. I'm running this in 8 GB of GPU

from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "bartowski/granite-3.0-8b-instruct-GGUF"
filename = "granite-3.0-8b-instruct-Q6_K_L.gguf"
tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

ERROR:

ValueError: Architecture granite not supported

I'm following this documentation from HF https://huggingface.co/docs/transformers/main/en/gguf