orel lavie

orel12's activity

• All
• Models
• Datasets
• Spaces
• Papers
• Collections
• Community
• Posts
• Upvotes
• Likes

New activity in jarrelscy/Mixtral-8x22B-Instruct-v0.1-GPTQ-4bit 6 months ago

I am running in vllm 0.4.1 with 4 x gpus 24gb (A10G 24gb) = 96gb and eager mode and I am still out of memory, how? it should fit (like 87gb vram)

1

#3 opened 6 months ago by orel12

KeyError: 'model.layers.45.block_sparse_moe.gate.g_idx'

5

#2 opened 6 months ago by tutu329