If true? Very impressive
You guys did better than OpenCodeInterpreter on their Gemma-7b finetune, im impressed, if this score is accurate.
Their model got a low score of 56.1 on human eval with Gemma
I used the evaluation method of bigcode-models-leaderboard for testing. It has some subtle differences from evalplus, but it will not affect the approximate results and rankings.
The loss for training gemma-7b using transformers is not stable (see https://huggingface.co/google/gemma-7b/discussions/46 and https://github.com/huggingface/transformers/issues/29219). OpenCodeInterpreter is a strong dataset and the difference is unlikely to be due to the data. I think it may be caused by not adjusting the training pipeline for Gemma.
@TechxGenus Congratulations on creating a highly performant code model! Your work is truly impressive. Could you please share the rough hyperparameters used for training the CodeGemma-7b model? Additionally, could you provide any insights or specific considerations to keep in mind regarding the training pipeline for the Gemma model during SFT? I appreciate your efforts and am looking forward to your response. Thank you!
Compared with training models such as codellama and deepseek-coder, I mainly made the following changes:
- lr = 5e-6 (training is very unstable when greater than 1e-5)
- max_grad_norm = 0.9
- Increase the number of warmup steps.
- Freeze embedding layer parameters for the first 0.1 epochs (Didn't do ablation, not sure if it really help).
If the loss suddenly increases by more than twice at the beginning or in the middle, the training results are likely to be unsatisfactory.
@aaabiao @TechxGenus You two are the example of ai companies everyone should follow. Sharing knowledge to advance ai. Rather than hoarding it. Google and openai should be ashamed. You make them look weak
@TechxGenus Thank you for sharing your valuable insights and modifications to the training approach! I greatly appreciate your prompt and detailed response. I will definitely try implementing your suggested methods in my own training experiments. Your advice could be a game-changer for improving model performance. Thanks again for your guidance!
@rombodawg My pleasure!!