Upload README.md

README.md

@@ -1,3 +1,28 @@
 ---
 license: apache-2.0
+tags:
+- MoE
+- merge
+- mergekit
+- Mistral
+- Microsoft/WizardLM-2-7B
 ---
+
+# WizardLM-2-4x7B-MoE
+
+This is an experimental MoE model made with Mergekit. It was made by combing four WizardLM-2-7B models using the random gate mode. 
+Please be sure to set experts per token to 4 for the best results! 
+Context length should be the same as Mistral-7B-Instruct-v0.1 (8k tokens).
+
+Mergekit config:
+```
+base_model: models/WizardLM-2-7B
+gate_mode: random
+dtype: float16
+experts_per_token: 4
+experts:
+  - source_model: models/WizardLM-2-7B
+  - source_model: models/WizardLM-2-7B
+  - source_model: models/WizardLM-2-7B
+  - source_model: models/WizardLM-2-7B
+```