---
base_model:
- ibm/merlinite-7b
- InferenceIllusionist/Magic-Dolphin-7b
- SanjiWatsuki/Kunoichi-DPO-v2-7B
- mlabonne/Monarch-7B
- bardsai/jaskier-7b-dpo-v6.1
library_name: transformers
tags:
- mergekit
- merge
license: apache-2.0
---
# Excalibur-7b
Update: A [fine-tuned version](https://huggingface.co/InferenceIllusionist/Excalibur-7b-DPO/) of this model is now publicly available, along with benchmark results. If you're looking for a more conversational, assistant-style exchange you won't want to miss it!
Image generated with Envoid's [Model9](https://huggingface.co/Envoid/model9) SDXL model
GGUFs can be found [here](https://huggingface.co/InferenceIllusionist/Excalibur-7b-GGUF)
Alternative GGUFs from [bartowski](https://huggingface.co/bartowski) can be found [here](https://huggingface.co/bartowski/Excalibur-7b-GGUF).
EXl2 can also be found [here](https://huggingface.co/bartowski/Excalibur-7b-exl2) again courtesy of [bartowski](https://huggingface.co/bartowski)!
### Performance Comparison
| Name | Avg. | ARC | HellaSwag | MMLU | TruthfulQA | Winogrande | GSM8K |
| ---- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |
| Excalibur-7b | 73.6 | 69.71 | 87.56 | 65.66 | 67.24 | 82.79 | 68.61 |
| Magic-Dolphin-7b | 67.48 | 65.78 | 85.61 | 64.64 | 58.01 | 79.64 | 51.18 |
| merlinite-7b | 64 | 63.65 | 84.52 | 64.91 | 50.15 | 79.72 | 41.09 |
[* Open LLM Leaderboard Dataset](https://huggingface.co/datasets/open-llm-leaderboard/details_InferenceIllusionist__Excalibur-7B)
### Methodology
[Magic-Dolphin-7b](https://huggingface.co/InferenceIllusionist/Magic-Dolphin-7b) was an unexpected surprise. Profoundly satisfied with it as a first attempt. For this follow-up I wanted to target the MMLU benchmark specifically.
The challenge this time was placing more weight on Merlinite-7b as an unknown quantity that hasn't been in the spotlight despite its novel LAB tuning method.
Excalibur-7b builds on past success and is the culmination of several learnings:
* Measuring KL-divergences for new quantization types brought a deeper understanding of benchmarking and assessing model performance
* This signifcantly sped up the testing process by using MMLU as a base, narrowing down over 10 candidate linear merges to 1: merliniteX-blockB1
* Reaching the limitations of linear merging necessitated a pivot to reviewing the viability of SLERP, DARE-TIES, and Passthrough methods
* Thus a competing candidate merge pool was tested between different merge algorithms. Once more the list was narrowed from 10 candidates to 1: merliniteX-blockF2
* merliniteX-blockF2 (SLERP of Magic-Dolphin-7B and jaskier-7b-dpo in unorthadox proportions) was originally planned for release earlier this week
* Instead -blockB1 and -blockF2 were merged and the results were placed head to head in a final round of tests. Ultimately a more conventional execution of SLERP showed the best results for the final step.
# Sample Question
# Bonus Question - Vision Capabilities
Requires additional [mistral-7b-mmproj-v1.5-Q4_1.gguf](https://huggingface.co/koboldcpp/mmproj/tree/main) file for vision functionality
Select up the gguf file of your choice in Kobold as usual, then make sure to choose the mmproj file above in the LLaVA mmproj field of the model submenu:
This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
## Merge Details
### Merge Method
This model was merged using the SLERP merge method.
### Models Merged
The following models were included in the merge:
* [ibm/merlinite-7b](https://huggingface.co/ibm/merlinite-7b)
* [InferenceIllusionist/Magic-Dolphin-7b](https://huggingface.co/InferenceIllusionist/Magic-Dolphin-7b)
* [SanjiWatsuki/Kunoichi-DPO-v2-7B](https://huggingface.co/SanjiWatsuki/Kunoichi-DPO-v2-7B)
* [mlabonne/Monarch-7B](https://huggingface.co/mlabonne/Monarch-7B)
* [bardsai/jaskier-7b-dpo-v6.1](https://huggingface.co/bardsai/jaskier-7b-dpo-v6.1)
### Configuration
The following YAML configurations were used to produce this model:
merliniteX-blockB1
```yaml
models:
- model: models/merlinite-7b
parameters:
weight: 1.0
- model: models/Kunoichi-DPO-v2-7B
parameters:
weight: 0.2
- model: models/jaskier-7b-dpo-v6.1
parameters:
weight: 0.6
- model: models/Monarch-7b
parameters:
weight: 0.4
merge_method: linear
dtype: float16
```
merliniteX-blockF2
```yaml
slices:
- sources:
- model: models/Magic-Dolphin-7b
layer_range: [0, 32]
- model: models/jaskier-7b-dpo-v6.1
layer_range: [0, 32]
merge_method: slerp
base_model: models/Magic-Dolphin-7b
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 0.5, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0.5, 0]
- value: 0.5 # fallback for rest of tensors
dtype: float16
```
merliniteX-blockH1 (Excalibur-7b)
```yaml
slices:
- sources:
- model: models/merliniteX-blockF2
layer_range: [0, 32]
- model: models/merliniteX-blockB1
layer_range: [0, 32]
merge_method: slerp
base_model: models/merliniteX-blockF2
parameters:
t:
- filter: self_attn
value: [1, 0.7, 0.3, 0.5, 0]
- filter: mlp
value: [0, 0.3, 0.7, 0.5, 1]
- value: 0.5 # fallback for rest of tensors
dtype: float16
```