Update README.md
Browse files
README.md
CHANGED
@@ -2,14 +2,79 @@
|
|
2 |
license: cc-by-nc-4.0
|
3 |
---
|
4 |
|
5 |
-
Don't mind
|
6 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
7 |
```
|
8 |
slices:
|
9 |
- sources:
|
10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
11 |
layer_range: [0, 48]
|
12 |
-
- model:
|
13 |
layer_range: [0, 48]
|
14 |
merge_method: slerp
|
15 |
base_model: Undi95/Mistral-11B-OpenOrcaPlatypus
|
@@ -30,6 +95,11 @@ parameters:
|
|
30 |
- value: 0.5 # fallback for rest of tensors
|
31 |
dtype: float16
|
32 |
```
|
|
|
|
|
|
|
|
|
|
|
33 |
|
34 |
hf-causal-experimental (pretrained=/content/drive/MyDrive/Mistral-11B-Test), limit: None, provide_description: False, num_fewshot: 0, batch_size: 4
|
35 |
| Task |Version| Metric |Value | |Stderr|
|
@@ -47,6 +117,16 @@ hf-causal-experimental (pretrained=/content/drive/MyDrive/Mistral-11B-Test), lim
|
|
47 |
|winogrande | 0|acc |0.7474|± |0.0122|
|
48 |
|
49 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
50 |
|
|
|
51 |
|
52 |
-
|
|
|
2 |
license: cc-by-nc-4.0
|
3 |
---
|
4 |
|
5 |
+
Don't mind this one at the moment, I need to finetune it for RP, it's just a test.
|
6 |
|
7 |
+
## Description
|
8 |
+
|
9 |
+
This repo contains fp16 files of Mistral-11B-OmniMix.
|
10 |
+
|
11 |
+
My goal for this model was only to make it score the highest possible with merge and layer toying, proving that:
|
12 |
+
- Benchmark are objective
|
13 |
+
- You should try a model yourself and don't go blindly to the highest rated one
|
14 |
+
- Merge/Layer toying CAN be usable to do better model (maybe?)
|
15 |
+
-
|
16 |
+
## Model used
|
17 |
+
- [Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca)
|
18 |
+
- [Mistral-7B-v0.1-Open-Platypus](akjindal53244/Mistral-7B-v0.1-Open-Platypus)
|
19 |
+
- [CollectiveCognition-v1.1-Mistral-7B](https://huggingface.co/teknium/CollectiveCognition-v1.1-Mistral-7B)
|
20 |
+
- [zephyr-7b-alpha](HuggingFaceH4/zephyr-7b-alpha)
|
21 |
+
-
|
22 |
+
|
23 |
+
|
24 |
+
## Prompt template: Alpaca or default
|
25 |
+
|
26 |
+
```
|
27 |
+
Below is an instruction that describes a task. Write a response that appropriately completes the request.
|
28 |
+
|
29 |
+
### Instruction:
|
30 |
+
{prompt}
|
31 |
+
|
32 |
+
### Response:
|
33 |
+
|
34 |
+
```
|
35 |
+
|
36 |
+
```
|
37 |
+
USER: <prompt>
|
38 |
+
ASSISTANT:
|
39 |
+
```
|
40 |
+
|
41 |
+
Or use any prompting system from one of the 4 source model, should work.
|
42 |
+
|
43 |
+
## The secret sauce
|
44 |
+
|
45 |
+
Mistral-11B-OpenOrcaPlatypus :
|
46 |
+
```
|
47 |
+
slices:
|
48 |
+
- sources:
|
49 |
+
- model: Open-Orca/Mistral-7B-OpenOrca
|
50 |
+
layer_range: [0, 24]
|
51 |
+
- sources:
|
52 |
+
- model: akjindal53244/Mistral-7B-v0.1-Open-Platypus
|
53 |
+
layer_range: [8, 32]
|
54 |
+
merge_method: passthrough
|
55 |
+
dtype: bfloat16
|
56 |
+
```
|
57 |
+
|
58 |
+
Mistral-11B-CC-Zephyr :
|
59 |
```
|
60 |
slices:
|
61 |
- sources:
|
62 |
+
- model: "/content/drive/MyDrive/CC-v1.1-7B-bf16"
|
63 |
+
layer_range: [0, 24]
|
64 |
+
- sources:
|
65 |
+
- model: "/content/drive/MyDrive/Zephyr-7B"
|
66 |
+
layer_range: [8, 32]
|
67 |
+
merge_method: passthrough
|
68 |
+
dtype: bfloat16
|
69 |
+
```
|
70 |
+
|
71 |
+
Mistral-11B-OmniMix :
|
72 |
+
```
|
73 |
+
slices:
|
74 |
+
- sources:
|
75 |
+
- model: Mistral-11B-OpenOrcaPlatypus
|
76 |
layer_range: [0, 48]
|
77 |
+
- model: Mistral-11B-CC-Zephyr
|
78 |
layer_range: [0, 48]
|
79 |
merge_method: slerp
|
80 |
base_model: Undi95/Mistral-11B-OpenOrcaPlatypus
|
|
|
95 |
- value: 0.5 # fallback for rest of tensors
|
96 |
dtype: float16
|
97 |
```
|
98 |
+
I use [mergekit](https://github.com/cg123/mergekit) for all the manipulation told here.
|
99 |
+
|
100 |
+
## Some scoring I done myself
|
101 |
+
|
102 |
+
This was named "Mistral-11B-TestBench11", keep that ine mind while looking trough this.
|
103 |
|
104 |
hf-causal-experimental (pretrained=/content/drive/MyDrive/Mistral-11B-Test), limit: None, provide_description: False, num_fewshot: 0, batch_size: 4
|
105 |
| Task |Version| Metric |Value | |Stderr|
|
|
|
117 |
|winogrande | 0|acc |0.7474|± |0.0122|
|
118 |
|
119 |
|
120 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/63ab1241ad514ca8d1430003/LggyIlV-oY7NbLwi7mnix.png)
|
121 |
+
|
122 |
+
This model seem to be the best out of my 3 latest try:
|
123 |
+
|
124 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/63ab1241ad514ca8d1430003/hnqNyljs5Y8JppuA_io8w.png)
|
125 |
+
|
126 |
+
You can find all the work I have done trying on this [Pastebin](https://pastebin.com/nHLCxQJv).
|
127 |
+
|
128 |
+
## Others
|
129 |
|
130 |
+
Special thanks to Sushi, [Henky](https://github.com/KoboldAI/KoboldAI-Client) for the machine he give me for big task, and [Charles Goddard](https://github.com/cg123) for his amazing tool.
|
131 |
|
132 |
+
If you want to support me, you can [here](https://ko-fi.com/undiai).
|