hiyouga

/

Llama-2-70b-AQLM-2Bit-QLoRA-function-calling

like 7

@prudant The training speed of AQLM fine-tuning is around 26.25s/example for a 70B model, so it requires ~15h to fine-tune the model over 2000 examples.

thanks!

can you share the command line and config files ?

https://github.com/hiyouga/LLaMA-Factory/blob/main/examples/qlora_single_gpu/aqlm.sh

why is this taking so little resources, compared to accelerate/fsdp_config.yaml ?
I am doing AlexWortega/miqu-1-70b-AQLM-2Bit-1x16-hf

(sorry i am a beginner)

Can you please tell how to format the data @hiyouga for training

can I convert mixtral 8x22 to AQLM and then train using this method on 2x3090?

@bittamer i think AQLM quant process require a lot of gpu computational power (more than 4 gpus for a couple of days running)

@bittamer I think FSDP+QLoRA should be more suitable for your case