databricks

/

dolly-v2-3b

like 287

Not sure. Did it download successfully? did you make any other modifications? You probably don't need to load in 8-bit with the 3b model, note.

I am following this demo https://www.dbdemos.ai/demo-notebooks.html?demoName=llm-dolly-chatbot

all the steps in the data preparation section were executed successfully.
in the prompt engineering section, I was not able to run this command : qa_chain = build_qa_chain() due to the exception mentioned above

I am loading it in 8-bit because it is mentioned as a note in the demo

am I getting it worng?

Try only changing the model name.
Or check if you're having trouble downloading the model - delete your HF cache and try again

what do you mean by changing the model name? I've already changed it dolly-3b instead of dolly-vb used in the demo? or should I use something else?

Right. I mean, only make that modification, not 8-bit or anything else. But I think you have a download problem.

@alaamigdady try this command to load the model in 8bit, note that the load_in_8bit is fed in via model_kwargs and not as a pipeline parameter. This is already fixed in a future release.

# Note: if you use dolly 12B or smaller model but a GPU with less than 24GB RAM, use 8bit. This requires %pip install bitsandbytes
instruct_pipeline = pipeline(model=model_name, trust_remote_code=True, device_map="auto", model_kwargs={'load_in_8bit': True})

Oh, you're not loading on a GPU. You should. I think you're out of OS memory here. Use a larger instance.