How did you create AWQ-quantized weights?
#5
by
nightdude
- opened
Could you share how you created the AWQ weights? Sorry, I'm new to AWQ (always used BNB). Thanks!
@casperhansen Can you share the config that you used to quantize this model, do you just use the default config? What is the data that you used for calibration?
Struggled a bit with quantisation, made some notes on what worked for me here: https://huggingface.co/mattmalcher/Llama-3-70B-Instruct-awq
This comment has been hidden