Provide learning code
#2
by
gihong99
- opened
I'm a student who learns and touches the LLM model. Can I get the information on the number of data and the learning code separately?
Hi, what exactly do you mean for the learning code?
If it is hyperparameters, this is it:
qLoRA finetune.
2 Epochs - Batch size of 1 on 4x 4090s.
Learning Rate - 0.00035
Scheduler - Cosine
Optimizer - adamw_bnb_8bit
LoRA rank - 16 | LoRA alpha - 32
lora_target_modules - [ - gate_proj - down_proj - up_proj - q_proj - v_proj - k_proj - o_proj]
For the dataset itself, that is private. Number of samples would be based on the model card, so 10% is 5.2K, total 52K. I listed the composition there.