File size: 6,918 Bytes
5800b7d b56b280 5800b7d 653f22f 1f02a31 653f22f ec90c12 653f22f ec90c12 653f22f ec90c12 653f22f 1f02a31 9fe4421 8543d2d ce6bbb2 214c36e 9fe4421 cf69b94 2363eb6 1c54f06 d2129b6 c7e6028 d2129b6 5800b7d 653f22f b73904f 92f765e b73904f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 |
---
base_model:
- beomi/Llama-3-KoEn-8B-Instruct-preview
- saltlux/Ko-Llama3-Luxia-8B
- cognitivecomputations/dolphin-2.9-llama3-8b
- NousResearch/Meta-Llama-3-8B
- nvidia/Llama3-ChatQA-1.5-8B
- aaditya/Llama3-OpenBioLLM-8B
- Danielbrdz/Barcenas-Llama3-8b-ORPO
- beomi/Llama-3-KoEn-8B-preview
- abacusai/Llama-3-Smaug-8B
- NousResearch/Meta-Llama-3-8B-Instruct
library_name: transformers
tags:
- mergekit
- merge
- llama
---
# YACHT-Llama-3-KoEn-8B
<a href="https://ibb.co/SyT5vj5"><img src="https://i.ibb.co/DWXzKJz/Screenshot-2024-05-07-at-3-04-45-AM.png" alt="Screenshot-2024-05-07-at-3-04-45-AM" border="0"></a>
🎵 *[JayLee LLMs Signature Tag] : ✍️ "I need a Jay Jay chat boy"* 🎵
✨ *Navigating the High Seas of Data: Crafting the Ultimate Yacht Insights with Merged LLMs* ✨
✨ *Aren’t you sometimes tired of just doing LLM & RAG & Normal Chat app? I'll show you a cool app soon integrating this my merged one(Tuned car). It wouldn't be fun if we only developed cars, so life is ultimately about driving cars and socializing with people.* ✨
🧨 *When using the Merge model for commercial purposes, a lot of care is needed. A mix of many models can be good, but it can also pose many risks.* 🧨
Thank you for visiting my page today!
Your donation makes me feel more free in Dev life. Instead, I will provide you with fun and useful software!!!
I haven't even released 0.001% of the software to you yet!!!
"Donation(ETH/USDT) : 0x8BB117dD4Cc0E19E5536ab211070c0dE039a85c0"
Can you borrow your computer power to merge heavy xtuner with my one cuz my com memory said that he is sick -> DM me!! (code ready)
```
Diff calculated for model.layers.13.self_attn.q_proj.weight
Diff calculated for model.layers.13.self_attn.k_proj.weight
Diff calculated for model.layers.13.self_attn.v_proj.weight
Diff calculated for model.layers.13.self_attn.o_proj.weight
Diff calculated for model.layers.13.mlp.gate_proj.weight
Diff calculated for model.layers.13.mlp.up_proj.weight
Diff calculated for model.layers.13.mlp.down_proj.weight
Diff calculated for model.layers.13.input_layernorm.weight
Diff calculated for model.layers.13.post_attention_layernorm.weight
Diff calculated for model.layers.14.self_attn.q_proj.weight
Diff calculated for model.layers.14.self_attn.k_proj.weight
Diff calculated for model.layers.14.self_attn.v_proj.weight
Diff calculated for model.layers.14.self_attn.o_proj.weight
Diff calculated for model.layers.14.mlp.gate_proj.weight
Diff calculated for model.layers.14.mlp.up_proj.weight
Diff calculated for model.layers.14.mlp.down_proj.weight
Diff calculated for model.layers.14.input_layernorm.weight
Diff calculated for model.layers.14.post_attention_layernorm.weight
(.venv) jaylee@lees-MacBook-Pro-2 merge % /opt/homebrew/Cellar/python@3.12/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
```
### Merge Method
This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method using [NousResearch/Meta-Llama-3-8B](https://huggingface.co/NousResearch/Meta-Llama-3-8B) as a base.
### Models Merged
The following models were included in the merge:
* [beomi/Llama-3-KoEn-8B-Instruct-preview](https://huggingface.co/beomi/Llama-3-KoEn-8B-Instruct-preview)
* [saltlux/Ko-Llama3-Luxia-8B](https://huggingface.co/saltlux/Ko-Llama3-Luxia-8B)
* [cognitivecomputations/dolphin-2.9-llama3-8b](https://huggingface.co/cognitivecomputations/dolphin-2.9-llama3-8b)
* [nvidia/Llama3-ChatQA-1.5-8B](https://huggingface.co/nvidia/Llama3-ChatQA-1.5-8B)
* [aaditya/Llama3-OpenBioLLM-8B](https://huggingface.co/aaditya/Llama3-OpenBioLLM-8B)
* [Danielbrdz/Barcenas-Llama3-8b-ORPO](https://huggingface.co/Danielbrdz/Barcenas-Llama3-8b-ORPO)
* [beomi/Llama-3-KoEn-8B-preview](https://huggingface.co/beomi/Llama-3-KoEn-8B-preview)
* [abacusai/Llama-3-Smaug-8B](https://huggingface.co/abacusai/Llama-3-Smaug-8B)
* [NousResearch/Meta-Llama-3-8B-Instruct](https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct)
### Configuration
The following YAML configuration was used to produce this model:
```yaml
models:
- model: NousResearch/Meta-Llama-3-8B
- model: NousResearch/Meta-Llama-3-8B-Instruct
parameters:
density: 0.60
weight: 0.25
- model: beomi/Llama-3-KoEn-8B-preview
parameters:
density: 0.55
weight: 0.2
- model: saltlux/Ko-Llama3-Luxia-8B
parameters:
density: 0.55
weight: 0.15
- model: beomi/Llama-3-KoEn-8B-Instruct-preview
parameters:
density: 0.55
weight: 0.15
- model: nvidia/Llama3-ChatQA-1.5-8B
parameters:
density: 0.55
weight: 0.1
- model: cognitivecomputations/dolphin-2.9-llama3-8b
parameters:
density: 0.55
weight: 0.05
- model: Danielbrdz/Barcenas-Llama3-8b-ORPO
parameters:
density: 0.55
weight: 0.05
- model: abacusai/Llama-3-Smaug-8B
parameters:
density: 0.55
weight: 0.05
- model: aaditya/Llama3-OpenBioLLM-8B
parameters:
density: 0.55
weight: 0.1
merge_method: dare_ties
base_model: NousResearch/Meta-Llama-3-8B
parameters:
int8_mask: true
dtype: bfloat16
```
### Test
<a href="https://ibb.co/tHkMB64"><img src="https://i.ibb.co/Zft6drV/Screenshot-2024-05-07-at-2-45-38-AM.png" alt="Screenshot-2024-05-07-at-2-45-38-AM" border="0"></a>
<a href="https://ibb.co/JtsN25B"><img src="https://i.ibb.co/V9qdD2j/Screenshot-2024-05-07-at-2-45-55-AM.png" alt="Screenshot-2024-05-07-at-2-45-55-AM" border="0"></a>
<a href="https://ibb.co/X8JhVMk"><img src="https://i.ibb.co/3zhDkQY/Screenshot-2024-05-07-at-2-46-11-AM.png" alt="Screenshot-2024-05-07-at-2-46-11-AM" border="0"></a>
<a href="https://ibb.co/XFZfvLm"><img src="https://i.ibb.co/qmMLPjb/Screenshot-2024-05-07-at-2-46-18-AM.png" alt="Screenshot-2024-05-07-at-2-46-18-AM" border="0"></a>
<a href="https://ibb.co/7Khf8B3"><img src="https://i.ibb.co/YhgCYJ5/Screenshot-2024-05-07-at-2-51-23-AM.png" alt="Screenshot-2024-05-07-at-2-51-23-AM" border="0"></a>
<a href="https://ibb.co/9tsDj40"><img src="https://i.ibb.co/sHyfr14/Screenshot-2024-05-07-at-2-46-48-AM.png" alt="Screenshot-2024-05-07-at-2-46-48-AM" border="0"></a>
<a href="https://ibb.co/z8FmLRz"><img src="https://i.ibb.co/HNCKMzj/Screenshot-2024-05-07-at-2-49-45-AM.png" alt="Screenshot-2024-05-07-at-2-49-45-AM" border="0"></a>
<a href="https://ibb.co/fvp9n9b"><img src="https://i.ibb.co/2NtSsSw/Screenshot-2024-05-07-at-3-20-20-PM.png" alt="Screenshot-2024-05-07-at-3-20-20-PM" border="0"></a>
<a href="https://ibb.co/fvp9n9b"><img src="https://i.ibb.co/2NtSsSw/Screenshot-2024-05-07-at-3-20-20-PM.png" alt="Screenshot-2024-05-07-at-3-20-20-PM" border="0"></a> |