RuntimeError: CUDA error
Getting this error when trying to run on Text-gen webui using runpod with 2xA6000.
RuntimeError: CUDA error: invalid device ordinal CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
I'm a total noob, so I don't know how to provide more info than this
Make sure you're using exllamav2 or exllamav2_hf as the loader in ooba. Also, you can specify a GPU memory split. You will want to enter a number a few GB less than the total for the first GPU. Since A6000s have 48GB VRAM, you can use something like 44,48 for your GPU split.
Thank you very much! That did the trick.