hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4 · Cuda failure 1 'invalid argument'

Hi all,

I tried to run the given model on the following host:

H100x8
Ubuntu 22.04
CPU x128
RAM x1.76 TB
Accelerators: ConnectX-7 x8, Hopper H100 (80 GB GPU memory) x8, NVSwitch Hopper x4
CUDA: 12.2
CUDA Docker Toolkit: properly installed

with the command:

docker run --gpus all --shm-size 1g -ti -p 8080:80 \
  -v hf_cache:/data \
  -e MODEL_ID=hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4 \
  -e NUM_SHARD=8 \
  -e QUANTIZE=awq \
  -e HF_TOKEN=$(cat ~/.cache/huggingface/token) \
  -e MAX_INPUT_LENGTH=4000 \
  -e MAX_TOTAL_TOKENS=4096 \
  ghcr.io/huggingface/text-generation-inference:2.2.0

Everything works as expected, but after trying to start the sharding, I always receive:

NCCL WARN Failed to execute operation Connect from rank 1, retcode 3
Cuda failure 1 'invalid argument'

Now I tried a different host yesterday with 8 x A100 which actually worked out of the box without this error, leading me to the question if someone experienced the same error on a system working with H100 or if it might be something host specific?

Best
Julian