Hugging Face's logo

bigscience
/
bloomz

Text Generation
Transformers
PyTorch
TensorBoard
Safetensors
bloom
Eval Results
text-generation-inference
Model card Files Metrics Community
56

Needed RAM for the full bloomz model?

#38
by luxianos - opened
Discussion
luxianos

Hi, how much RAM memory would be needed to load the full bloomz model? I'm trying different aws instance types configurations but at least 768GB still is not enough.

gus07ven

@luxianos did you find a solution to this? I'm trying to deploy it on AWS as well and keep getting errors.

Muennighoff
 BigScience Workshop org

I generally use 8 * 80GB A100s, which works fine with 8-way pipeline parallelism, so 768GB should be enough, maybe you're not loading it in bfloat16?

luxianos

@luxianos did you find a solution to this? I'm trying to deploy it on AWS as well and keep getting errors.

I found around 800-850GB was enough

Sign up or log in to comment