Computer RAM requirements
#12
by
SVBilenko
- opened
Can you tell me how much computer RAM is needed to deploy this model? My attempt to deploy the model on a computer with 64 gigabytes of RAM ended in failure due to lack of memory.
i added 40gb of swap and it took up 22gb of it, perhaps adding bf16?
would a ggml model run better on cpus?
@pseudon how to add the 40gb swap? am a bit of a noob sorry
how to add the 40gb swap? am a bit of a noob sorry
sudo dd if=/dev/zero of=/.swap bs=16777216 count=2560
sudo mkswap /.swap
sudo swapon -v /.swap
Closing this issue as we added a hardware requirements section here and we have a ggml implementation at starcoder.cpp
loubnabnl
changed discussion status to
closed