how to load this model to multiple gpus?
#18
by
yijiu
- opened
I have 4 GPU A10G each one's memory is 24g, but this model needs 26g memory, so I need to load it into at least 2 GPUs at the same time, and I am a beginner at embedding, so I don't know how to do that, could anyone tell me? thank you!
You can use the below code:
model = AutoModel.from_pretrained('Alibaba-NLP/gte-Qwen1.5-7B-instruct', trust_remote_code=True, device_map='auto')