请问A10单卡24G,能支持该模型的预测功能吗？
Suppose we use single A10 with 24G gpu memory, can we support the model's prediction function?
How about the fine tuning of the Model? how many A10 is needed?
Hi, the parameters of the XuanYuan-70B base-model are 70B. Hence, it occupies 129G of VRAM and requires at least two 80G VRAM GPUs during model inference.
For fine-tuning, it needs to be categorized:
If it's full-parameter fine-tuning, at least four machines with 8 A100/A800 cards each are needed;
If fine-tuning with Lora, etc., one machine with 8 A100/A800 cards will suffice.
Additionally, the A10 GPU is more suited for inference and not as efficient for training.
Hi, the parameters of the XuanYuan-70B base-model are 70B. Hence, it occupies 129G of VRAM and requires at least two 80G VRAM GPUs during model inference.
For fine-tuning, it needs to be categorized:
If it's full-parameter fine-tuning, at least four machines with 8 A100/A800 cards each are needed;
If fine-tuning with Lora, etc., one machine with 8 A100/A800 cards will suffice.
Additionally, the A10 GPU is more suited for inference and not as efficient for training.
OK,Thank you very much for the very detailed reply