It works!!!

#1
by HoangHa - opened

Congratulation. You are amazing. Are you using linear or other kind of merge method?

Yeah, I want to add, great job man, it's people like you that push fronts forward, RESPECT!

deleted

Even if this doesn't work, what a brilliant idea. Combining various Mistrals together in a MOE never even occurred to me.

I feel this could be the better way than finetuning a MoE. We don't really wanna experts be the same.

deleted
This comment has been hidden

What an amusing concept. :D

Thanks for all the interest! I've made the script for making this kind of merge public, and wrote a bit about the reasoning and methodology. See my comment here.

Thank you @chargoddard and @Undi95 . You guys are absolute genius

Sign up or log in to comment