Update README.md

README.md

@@ -2,14 +2,79 @@
 license: cc-by-nc-4.0
 ---
 
-Don't mind those at the moment, I need to finetune them for RP, it's just some tests.
+Don't mind this one at the moment, I need to finetune it for RP, it's just a test.
 
+## Description
+
+This repo contains fp16 files of Mistral-11B-OmniMix.
+
+My goal for this model was only to make it score the highest possible with merge and layer toying, proving that:
+- Benchmark are objective
+- You should try a model yourself and don't go blindly to the highest rated one
+- Merge/Layer toying CAN be usable to do better model (maybe?)
+- 
+## Model used
+- [Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca)
+- [Mistral-7B-v0.1-Open-Platypus](akjindal53244/Mistral-7B-v0.1-Open-Platypus)
+- [CollectiveCognition-v1.1-Mistral-7B](https://huggingface.co/teknium/CollectiveCognition-v1.1-Mistral-7B)
+- [zephyr-7b-alpha](HuggingFaceH4/zephyr-7b-alpha)
+- 
+
+
+## Prompt template: Alpaca or default
+
+```
+Below is an instruction that describes a task. Write a response that appropriately completes the request.
+
+### Instruction:
+{prompt}
+
+### Response:
+
+```
+
+```
+USER: <prompt>
+ASSISTANT:
+```
+
+Or use any prompting system from one of the 4 source model, should work.
+
+## The secret sauce
+
+Mistral-11B-OpenOrcaPlatypus :
+```
+slices:
+  - sources:
+    - model: Open-Orca/Mistral-7B-OpenOrca
+      layer_range: [0, 24]
+  - sources:
+    - model: akjindal53244/Mistral-7B-v0.1-Open-Platypus
+      layer_range: [8, 32]
+merge_method: passthrough
+dtype: bfloat16
+```
+
+Mistral-11B-CC-Zephyr :
 ```
 slices:
   - sources:
-      - model: Undi95/Mistral-11B-OpenOrcaPlatypus
+    - model: "/content/drive/MyDrive/CC-v1.1-7B-bf16"
+      layer_range: [0, 24]
+  - sources:
+    - model: "/content/drive/MyDrive/Zephyr-7B"
+      layer_range: [8, 32]
+merge_method: passthrough
+dtype: bfloat16
+```
+
+Mistral-11B-OmniMix :
+```
+slices:
+  - sources:
+      - model: Mistral-11B-OpenOrcaPlatypus
         layer_range: [0, 48]
-      - model: "/content/drive/MyDrive/Mistral-11B-CC-Zephyr"
+      - model: Mistral-11B-CC-Zephyr
         layer_range: [0, 48]
 merge_method: slerp
 base_model: Undi95/Mistral-11B-OpenOrcaPlatypus
@@ -30,6 +95,11 @@ parameters:
     - value: 0.5 # fallback for rest of tensors
 dtype: float16
 ```
+I use [mergekit](https://github.com/cg123/mergekit) for all the manipulation told here.
+
+## Some scoring I done myself
+
+This was named "Mistral-11B-TestBench11", keep that ine mind while looking trough this.
 
 hf-causal-experimental (pretrained=/content/drive/MyDrive/Mistral-11B-Test), limit: None, provide_description: False, num_fewshot: 0, batch_size: 4
 |    Task     |Version| Metric |Value |   |Stderr|
@@ -47,6 +117,16 @@ hf-causal-experimental (pretrained=/content/drive/MyDrive/Mistral-11B-Test), lim
 |winogrande   |      0|acc     |0.7474|±  |0.0122|
 
 
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/63ab1241ad514ca8d1430003/LggyIlV-oY7NbLwi7mnix.png)
+
+This model seem to be the best out of my 3 latest try:
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/63ab1241ad514ca8d1430003/hnqNyljs5Y8JppuA_io8w.png)
+
+You can find all the work I have done trying on this [Pastebin](https://pastebin.com/nHLCxQJv).
+
+## Others
 
+Special thanks to Sushi, [Henky](https://github.com/KoboldAI/KoboldAI-Client) for the machine he give me for big task, and [Charles Goddard](https://github.com/cg123) for his amazing tool.
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/63ab1241ad514ca8d1430003/8f-rAHIfN1ZuW4HqkzYz-.png)
+If you want to support me, you can [here](https://ko-fi.com/undiai).