openlm-research

/

open_llama_3b

like 154

Follow

OpenLM Research 182

I kept seeing this error:

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

I doubt it's because of out of memory, because I also see the following:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 5.79 GiB total capacity; 4.66 GiB already allocated; 40.94 MiB free; 4.75 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

which is also weird, because there is enough free memory for allocate... anyone see the same error?