metadata
language:
- en
license: unknown
tags:
- frankenmerge
- 110b
pipeline_tag: conversational
model-index:
- name: BigWeave-v20-110b
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 68.17
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=llmixer/BigWeave-v20-110b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 88.54
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=llmixer/BigWeave-v20-110b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 70.51
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=llmixer/BigWeave-v20-110b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 62.47
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=llmixer/BigWeave-v20-110b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 82.08
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=llmixer/BigWeave-v20-110b
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 36.39
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=llmixer/BigWeave-v20-110b
name: Open LLM Leaderboard
BigWeave v20 110b
![](https://cdn-uploads.huggingface.co/production/uploads/65a6db055c58475cf9e6def1/4CbbAN-X7ZWj702JrcCGH.png)
The BigWeave models aim to experimentally identify merge settings for increasing model performance. The version number merely tracks various attempts and is not a quality indicator. Only results demonstrating good performance are retained and shared.
Prompting Format
Mistral, Vicuna and Alpaca.
Merge process
This is a merge of 152334H/miqu-1-70b-sf and lizpreciatior/lzlv_70b_fp16_hf. By conducting exl2 measurements, we identify the least important layers of lzlv. These least important layers are extended with layers in-between to create longer series of consecutive layers. These slices are then inserted into miqu.
Merge configuration:
slices:
- sources:
- model: 152334H/miqu-1-70b-sf
layer_range: [0, 1]
- model: lizpreciatior/lzlv_70b_fp16_hf
layer_range: [0, 1]
parameters:
weight: 0
- sources:
- model: 152334H/miqu-1-70b-sf
layer_range: [1,26]
- sources:
- model: lizpreciatior/lzlv_70b_fp16_hf
layer_range: [9,44]
- sources:
- model: 152334H/miqu-1-70b-sf
layer_range: [27,52]
- sources:
- model: lizpreciatior/lzlv_70b_fp16_hf
layer_range: [45,60]
- sources:
- model: 152334H/miqu-1-70b-sf
layer_range: [53,79]
- sources:
- model: 152334H/miqu-1-70b-sf
layer_range: [79, 80]
- model: lizpreciatior/lzlv_70b_fp16_hf
layer_range: [79, 80]
parameters:
weight: 0
merge_method: linear
parameters:
weight: 1.0
dtype: float16
tokenizer_source: 152334H/miqu-1-70b-sf
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
Metric | Value |
---|---|
Avg. | 68.03 |
AI2 Reasoning Challenge (25-Shot) | 68.17 |
HellaSwag (10-Shot) | 88.54 |
MMLU (5-Shot) | 70.51 |
TruthfulQA (0-shot) | 62.47 |
Winogrande (5-shot) | 82.08 |
GSM8k (5-shot) | 36.39 |