Upload folder using huggingface_hub
Browse files
README.md
ADDED
@@ -0,0 +1,252 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
base_model:
|
3 |
+
- nbeerbower/llama-3-stella-8B
|
4 |
+
- Hastagaras/llama-3-8b-okay
|
5 |
+
- nbeerbower/llama-3-gutenberg-8B
|
6 |
+
- openchat/openchat-3.6-8b-20240522
|
7 |
+
- Kukedlc/NeuralLLaMa-3-8b-DT-v0.1
|
8 |
+
- cstr/llama3-8b-spaetzle-v20
|
9 |
+
- mlabonne/ChimeraLlama-3-8B-v3
|
10 |
+
- flammenai/Mahou-1.1-llama3-8B
|
11 |
+
- KingNish/KingNish-Llama3-8b
|
12 |
+
license: other
|
13 |
+
tags:
|
14 |
+
- merge
|
15 |
+
- mergekit
|
16 |
+
- lazymergekit
|
17 |
+
- autoquant
|
18 |
+
- exl2
|
19 |
+
model-index:
|
20 |
+
- name: Daredevil-8B
|
21 |
+
results:
|
22 |
+
- task:
|
23 |
+
type: text-generation
|
24 |
+
name: Text Generation
|
25 |
+
dataset:
|
26 |
+
name: AI2 Reasoning Challenge (25-Shot)
|
27 |
+
type: ai2_arc
|
28 |
+
config: ARC-Challenge
|
29 |
+
split: test
|
30 |
+
args:
|
31 |
+
num_few_shot: 25
|
32 |
+
metrics:
|
33 |
+
- type: acc_norm
|
34 |
+
value: 68.86
|
35 |
+
name: normalized accuracy
|
36 |
+
source:
|
37 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/Daredevil-8B
|
38 |
+
name: Open LLM Leaderboard
|
39 |
+
- task:
|
40 |
+
type: text-generation
|
41 |
+
name: Text Generation
|
42 |
+
dataset:
|
43 |
+
name: HellaSwag (10-Shot)
|
44 |
+
type: hellaswag
|
45 |
+
split: validation
|
46 |
+
args:
|
47 |
+
num_few_shot: 10
|
48 |
+
metrics:
|
49 |
+
- type: acc_norm
|
50 |
+
value: 84.5
|
51 |
+
name: normalized accuracy
|
52 |
+
source:
|
53 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/Daredevil-8B
|
54 |
+
name: Open LLM Leaderboard
|
55 |
+
- task:
|
56 |
+
type: text-generation
|
57 |
+
name: Text Generation
|
58 |
+
dataset:
|
59 |
+
name: MMLU (5-Shot)
|
60 |
+
type: cais/mmlu
|
61 |
+
config: all
|
62 |
+
split: test
|
63 |
+
args:
|
64 |
+
num_few_shot: 5
|
65 |
+
metrics:
|
66 |
+
- type: acc
|
67 |
+
value: 69.24
|
68 |
+
name: accuracy
|
69 |
+
source:
|
70 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/Daredevil-8B
|
71 |
+
name: Open LLM Leaderboard
|
72 |
+
- task:
|
73 |
+
type: text-generation
|
74 |
+
name: Text Generation
|
75 |
+
dataset:
|
76 |
+
name: TruthfulQA (0-shot)
|
77 |
+
type: truthful_qa
|
78 |
+
config: multiple_choice
|
79 |
+
split: validation
|
80 |
+
args:
|
81 |
+
num_few_shot: 0
|
82 |
+
metrics:
|
83 |
+
- type: mc2
|
84 |
+
value: 59.89
|
85 |
+
source:
|
86 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/Daredevil-8B
|
87 |
+
name: Open LLM Leaderboard
|
88 |
+
- task:
|
89 |
+
type: text-generation
|
90 |
+
name: Text Generation
|
91 |
+
dataset:
|
92 |
+
name: Winogrande (5-shot)
|
93 |
+
type: winogrande
|
94 |
+
config: winogrande_xl
|
95 |
+
split: validation
|
96 |
+
args:
|
97 |
+
num_few_shot: 5
|
98 |
+
metrics:
|
99 |
+
- type: acc
|
100 |
+
value: 78.45
|
101 |
+
name: accuracy
|
102 |
+
source:
|
103 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/Daredevil-8B
|
104 |
+
name: Open LLM Leaderboard
|
105 |
+
- task:
|
106 |
+
type: text-generation
|
107 |
+
name: Text Generation
|
108 |
+
dataset:
|
109 |
+
name: GSM8k (5-shot)
|
110 |
+
type: gsm8k
|
111 |
+
config: main
|
112 |
+
split: test
|
113 |
+
args:
|
114 |
+
num_few_shot: 5
|
115 |
+
metrics:
|
116 |
+
- type: acc
|
117 |
+
value: 73.54
|
118 |
+
name: accuracy
|
119 |
+
source:
|
120 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/Daredevil-8B
|
121 |
+
name: Open LLM Leaderboard
|
122 |
+
---
|
123 |
+
|
124 |
+
# Daredevil-8B
|
125 |
+
|
126 |
+
![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/gFEhcIDSKa3AWpkNfH91q.jpeg)
|
127 |
+
|
128 |
+
Daredevil-8B is a mega-merge designed to maximize MMLU. On 27 May 24, it is the Llama 3 8B model with the **highest MMLU score**.
|
129 |
+
From my experience, a high MMLU score is all you need with Llama 3 models.
|
130 |
+
|
131 |
+
It is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
|
132 |
+
* [nbeerbower/llama-3-stella-8B](https://huggingface.co/nbeerbower/llama-3-stella-8B)
|
133 |
+
* [Hastagaras/llama-3-8b-okay](https://huggingface.co/Hastagaras/llama-3-8b-okay)
|
134 |
+
* [nbeerbower/llama-3-gutenberg-8B](https://huggingface.co/nbeerbower/llama-3-gutenberg-8B)
|
135 |
+
* [openchat/openchat-3.6-8b-20240522](https://huggingface.co/openchat/openchat-3.6-8b-20240522)
|
136 |
+
* [Kukedlc/NeuralLLaMa-3-8b-DT-v0.1](https://huggingface.co/Kukedlc/NeuralLLaMa-3-8b-DT-v0.1)
|
137 |
+
* [cstr/llama3-8b-spaetzle-v20](https://huggingface.co/cstr/llama3-8b-spaetzle-v20)
|
138 |
+
* [mlabonne/ChimeraLlama-3-8B-v3](https://huggingface.co/mlabonne/ChimeraLlama-3-8B-v3)
|
139 |
+
* [flammenai/Mahou-1.1-llama3-8B](https://huggingface.co/flammenai/Mahou-1.1-llama3-8B)
|
140 |
+
* [KingNish/KingNish-Llama3-8b](https://huggingface.co/KingNish/KingNish-Llama3-8b)
|
141 |
+
|
142 |
+
Thanks to nbeerbower, Hastagaras, openchat, Kukedlc, cstr, flammenai, and KingNish for their merges. Special thanks to Charles Goddard and Arcee.ai for MergeKit.
|
143 |
+
|
144 |
+
## π Applications
|
145 |
+
|
146 |
+
You can use it as an improved version of [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct).
|
147 |
+
|
148 |
+
This is a censored model. For an uncensored version, see [mlabonne/Daredevil-8B-abliterated](https://huggingface.co/mlabonne/Daredevil-8B-abliterated).
|
149 |
+
|
150 |
+
Tested on LM Studio using the "Llama 3" preset.
|
151 |
+
|
152 |
+
## β‘ Quantization
|
153 |
+
|
154 |
+
* **GGUF**: https://huggingface.co/mlabonne/Daredevil-8B-GGUF
|
155 |
+
|
156 |
+
## π Evaluation
|
157 |
+
|
158 |
+
### Open LLM Leaderboard
|
159 |
+
|
160 |
+
Daredevil-8B is the best-performing 8B model on the Open LLM Leaderboard in terms of MMLU score (27 May 24).
|
161 |
+
|
162 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/xFKhGdSaIxL9_tcJPhM5w.png)
|
163 |
+
|
164 |
+
### Nous
|
165 |
+
|
166 |
+
Daredevil-8B is the best-performing 8B model on Nous' benchmark suite (evaluation performed using [LLM AutoEval](https://github.com/mlabonne/llm-autoeval), 27 May 24). See the entire leaderboard [here](https://huggingface.co/spaces/mlabonne/Yet_Another_LLM_Leaderboard).
|
167 |
+
|
168 |
+
| Model | Average | AGIEval | GPT4All | TruthfulQA | Bigbench |
|
169 |
+
|---|---:|---:|---:|---:|---:|
|
170 |
+
| [**mlabonne/Daredevil-8B**](https://huggingface.co/mlabonne/Daredevil-8B) [π](https://gist.github.com/mlabonne/080f9c5f153ea57a7ab7d932cf896f21) | **55.87** | **44.13** | **73.52** | **59.05** | **46.77** |
|
171 |
+
| [mlabonne/Daredevil-8B-abliterated](https://huggingface.co/mlabonne/Daredevil-8B-abliterated) [π](https://gist.github.com/mlabonne/32cdd8460804662c856bcb2a20acd49e) | 55.06 | 43.29 | 73.33 | 57.47 | 46.17 |
|
172 |
+
| [mlabonne/Llama-3-8B-Instruct-abliterated-dpomix](https://huggingface.co/mlabonne/Llama-3-8B-Instruct-abliterated-dpomix) [π](https://gist.github.com/mlabonne/d711548df70e2c04771cc68ab33fe2b9) | 52.26 | 41.6 | 69.95 | 54.22 | 43.26 |
|
173 |
+
| [meta-llama/Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) [π](https://gist.github.com/mlabonne/8329284d86035e6019edb11eb0933628) | 51.34 | 41.22 | 69.86 | 51.65 | 42.64 |
|
174 |
+
| [failspy/Meta-Llama-3-8B-Instruct-abliterated-v3](https://huggingface.co/failspy/Meta-Llama-3-8B-Instruct-abliterated-v3) [π](https://gist.github.com/mlabonne/f46cce0262443365e4cce2b6fa7507fc) | 51.21 | 40.23 | 69.5 | 52.44 | 42.69 |
|
175 |
+
| [mlabonne/OrpoLlama-3-8B](https://huggingface.co/mlabonne/OrpoLlama-3-8B) [π](https://gist.github.com/mlabonne/22896a1ae164859931cc8f4858c97f6f) | 48.63 | 34.17 | 70.59 | 52.39 | 37.36 |
|
176 |
+
| [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) [π](https://gist.github.com/mlabonne/616b6245137a9cfc4ea80e4c6e55d847) | 45.42 | 31.1 | 69.95 | 43.91 | 36.7 |
|
177 |
+
|
178 |
+
## π³ Model family tree
|
179 |
+
|
180 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/ekwRGgnjzEOyprT8sEBFt.png)
|
181 |
+
|
182 |
+
## 𧩠Configuration
|
183 |
+
|
184 |
+
```yaml
|
185 |
+
models:
|
186 |
+
- model: NousResearch/Meta-Llama-3-8B
|
187 |
+
# No parameters necessary for base model
|
188 |
+
- model: nbeerbower/llama-3-stella-8B
|
189 |
+
parameters:
|
190 |
+
density: 0.6
|
191 |
+
weight: 0.16
|
192 |
+
- model: Hastagaras/llama-3-8b-okay
|
193 |
+
parameters:
|
194 |
+
density: 0.56
|
195 |
+
weight: 0.1
|
196 |
+
- model: nbeerbower/llama-3-gutenberg-8B
|
197 |
+
parameters:
|
198 |
+
density: 0.6
|
199 |
+
weight: 0.18
|
200 |
+
- model: openchat/openchat-3.6-8b-20240522
|
201 |
+
parameters:
|
202 |
+
density: 0.56
|
203 |
+
weight: 0.12
|
204 |
+
- model: Kukedlc/NeuralLLaMa-3-8b-DT-v0.1
|
205 |
+
parameters:
|
206 |
+
density: 0.58
|
207 |
+
weight: 0.18
|
208 |
+
- model: cstr/llama3-8b-spaetzle-v20
|
209 |
+
parameters:
|
210 |
+
density: 0.56
|
211 |
+
weight: 0.08
|
212 |
+
- model: mlabonne/ChimeraLlama-3-8B-v3
|
213 |
+
parameters:
|
214 |
+
density: 0.56
|
215 |
+
weight: 0.08
|
216 |
+
- model: flammenai/Mahou-1.1-llama3-8B
|
217 |
+
parameters:
|
218 |
+
density: 0.55
|
219 |
+
weight: 0.05
|
220 |
+
- model: KingNish/KingNish-Llama3-8b
|
221 |
+
parameters:
|
222 |
+
density: 0.55
|
223 |
+
weight: 0.05
|
224 |
+
merge_method: dare_ties
|
225 |
+
base_model: NousResearch/Meta-Llama-3-8B
|
226 |
+
dtype: bfloat16
|
227 |
+
```
|
228 |
+
|
229 |
+
## π» Usage
|
230 |
+
|
231 |
+
```python
|
232 |
+
!pip install -qU transformers accelerate
|
233 |
+
|
234 |
+
from transformers import AutoTokenizer
|
235 |
+
import transformers
|
236 |
+
import torch
|
237 |
+
|
238 |
+
model = "mlabonne/Daredevil-8B"
|
239 |
+
messages = [{"role": "user", "content": "What is a large language model?"}]
|
240 |
+
|
241 |
+
tokenizer = AutoTokenizer.from_pretrained(model)
|
242 |
+
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
243 |
+
pipeline = transformers.pipeline(
|
244 |
+
"text-generation",
|
245 |
+
model=model,
|
246 |
+
torch_dtype=torch.bfloat16,
|
247 |
+
device_map="auto",
|
248 |
+
)
|
249 |
+
|
250 |
+
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
|
251 |
+
print(outputs[0]["generated_text"])
|
252 |
+
```
|