Multi GPU inference issue
#39
by
eastwind
- opened
When inferencing over multiple gpus I get gibberish unless I pass use_caching=False in the model.generate function. Not sure why this happens.
I had issues running it turns out you need over 250GB of Disk Volume first for weights, packages, and os dependencies