Runtime error Space evicted, storage limit exceeded (50G)
#5
by
Ibrahemqasim
- opened
I already used:
model = AutoModelForCausalLM.from_pretrained( "inception-mbzuai/jais-13b-chat", torch_dtype=torch.float16, trust_remote_code=True)
but still have storage issue
Try to quantize the model in lower bit precision e.g. 8 bit or 4 bit. The following code works with 128GB main memory and a 32GB GPU.
from peft import prepare_model_for_kbit_training
model = AutoModelForCausalLM.from_pretrained("inception-mbzuai/jais-13b-chat", load_in_8bit=True, device_map="auto", trust_remote_code=True)
model = prepare_model_for_kbit_training(model)