|
--- |
|
license: llama2 |
|
tags: |
|
- merge |
|
- mergekit |
|
--- |
|
# BETTER THAN GOLIATH?! |
|
I've merged [Euryale-lora that I made](https://huggingface.co/ChuckMcSneed/Euryale-1.3-L2-70B-LORA) with [Xwin](https://huggingface.co/Xwin-LM/Xwin-LM-70B-V0.1) and then merged it with itself in [goliath-style merge](/config.yml) using [mergekit](https://github.com/arcee-ai/mergekit). The resulting model performs better than [goliath](https://huggingface.co/alpindale/goliath-120b) on my tests(note: performance on tests is not necessarily performance in practice). Test it, have fun with it. This is a sister model of [Premerge-EX-EX-123B](https://huggingface.co/ChuckMcSneed/Premerge-EX-EX-123B). |
|
# Prompt format |
|
Alpaca. |
|
# Ideas behind it |
|
Since the creation of Goliath I was wondering if it was possible to make something even better. I've tried linear, passthrough, SLERP, TIES-merging models, but I could not recreate the greatness of goliath, at least not in a way that I liked in practical use. I knew about the existence of LORAs but I didn't know how well they performed. I created a model named [Gembo](https://huggingface.co/ChuckMcSneed/Gembo-v1-70b) by merging a shitton of LORAs together, and surprisingly it worked! In fact it worked so well that it was the best model on my benchmarks until now. When I found a tool named [LORD](https://github.com/thomasgauthier/LoRD), which can extract LORA from any model, I knew I could do something even better. |
|
|
|
I've extracted LORA from Euryale, then from Xwin and began testing. Merging Euryale-lora to Xwin and the other way around, created better models, which outperformed their parents: |
|
|
|
|Name |Quant|Size|B |C |D |S |P |total|BCD|SP | |
|
|-------------------------------------|-----|----|---|---|---|---|----|-----|---|-----| |
|
|Sao10K/Euryale-1.3-L2-70B |Q6_K |70B |0 |2 |0 |3 |5 |10 |2 |8 | |
|
|Sao10K/Euryale-1.3-L2-70B+xwin-lora |Q6_K |70B |2 |2 |1 |5.5|5.5 |16 |5 |11 | |
|
|Xwin-LM/Xwin-LM-70B-V0.1 |Q6_K |70B |0 |1 |2 |5.5|5.25|13.75|3 |10.75| |
|
|Xwin-LM/Xwin-LM-70B-V0.1+euryale-lora|Q6_K |70B |3 |2 |2 |6 |5 |18 |7 |11 | |
|
|
|
Results seemed promising, so I continued testing, merging it in goliath-like way in different orders(EX=Euryale+LORAXwin; XE=Xwin+LORAEuryale). The results were even more surprising: |
|
|
|
|Name |Quant|Size|B |C |D |S |P |total|BCD|SP | |
|
|-------------------------------------|-----|----|---|---|---|---|----|-----|---|-----| |
|
|alpindale/goliath-120b |Q6_K |120B|3 |2 |1 |6 |6 |18 |6 |12 | |
|
|ChuckMcSneed/Premerge-EX-EX-123B |Q6_K |123B|2 |2 |1.5|7.25|6 |18.75|5.5|13.25| |
|
|ChuckMcSneed/Premerge-EX-XE-123B |Q6_K |123B|2 |2 |2 |5.75|6 |17.75|6 |11.75| |
|
|ChuckMcSneed/Premerge-XE-EX-123B |Q6_K |123B|2 |2 |2.5|6.75|5.5 |18.75|6.5|12.25| |
|
|ChuckMcSneed/Premerge-XE-XE-123B(this model) |Q6_K |123B|3 |2 |2.5|7.25|5.25|20 |7.5|12.5 | |
|
|Sao10K/Euryale-1.3-L2-70B+xwin-lora |Q6_K |70B |2 |2 |1 |5.5|5.5 |16 |5 |11 | |
|
|Xwin-LM/Xwin-LM-70B-V0.1+euryale-lora|Q6_K |70B |3 |2 |2 |6 |5 |18 |7 |11 | |
|
|
|
Contrary to my expectations, merging two different models was suboptimal in this case. Selfmerge of Euryale-LORAXwin did beat all of the other merges on SP tests(creative writing), making it the highest scoring model on those tests that I've tested so far, and selfmerge of Xwin-LORAEuryale(this model) had highest score overall. |
|
# What it means |
|
Potentially in the future we can get better models by controlled merging of LORAs. |
|
# Benchmarks |
|
### NeoEvalPlusN |
|
[My meme benchmark.](https://huggingface.co/datasets/ChuckMcSneed/NeoEvalPlusN_benchmark) |
|
|Name |Quant|Size|B |C |D |S |P |total|BCD|SP | |
|
|-------------------------------------|-----|----|---|---|---|---|----|-----|---|-----| |
|
|alpindale/goliath-120b |Q6_K |120B|3 |2 |1 |6 |6 |18 |6 |12 | |
|
|ChuckMcSneed/Premerge-EX-EX-123B |Q6_K |123B|2 |2 |1.5|7.25|6 |18.75|5.5|13.25| |
|
|ChuckMcSneed/Premerge-EX-XE-123B |Q6_K |123B|2 |2 |2 |5.75|6 |17.75|6 |11.75| |
|
|ChuckMcSneed/Premerge-XE-EX-123B |Q6_K |123B|2 |2 |2.5|6.75|5.5 |18.75|6.5|12.25| |
|
|ChuckMcSneed/Premerge-XE-XE-123B(this model) |Q6_K |123B|3 |2 |2.5|7.25|5.25|20 |7.5|12.5 | |
|
|Sao10K/Euryale-1.3-L2-70B |Q6_K |70B |0 |2 |0 |3 |5 |10 |2 |8 | |
|
|Sao10K/Euryale-1.3-L2-70B+xwin-lora |Q6_K |70B |2 |2 |1 |5.5|5.5 |16 |5 |11 | |
|
|Xwin-LM/Xwin-LM-70B-V0.1 |Q6_K |70B |0 |1 |2 |5.5|5.25|13.75|3 |10.75| |
|
|Xwin-LM/Xwin-LM-70B-V0.1+euryale-lora|Q6_K |70B |3 |2 |2 |6 |5 |18 |7 |11 | |
