--- license: apache-2.0 tags: - moe - frankenmoe - merge - mergekit - lazymergekit - jsfs11/RandomMergeNoNormWEIGHTED-7B-DARETIES - senseable/WestLake-7B-v2 - mlabonne/OmniBeagle-7B - vanillaOVO/supermario_v3 base_model: - jsfs11/RandomMergeNoNormWEIGHTED-7B-DARETIES - senseable/WestLake-7B-v2 - mlabonne/OmniBeagle-7B - vanillaOVO/supermario_v3 --- # MixtureofMerges-MoE-4x7b-v3 MixtureofMerges-MoE-4x7b-v3 is a Mixure of Experts (MoE) made with the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing): * [jsfs11/RandomMergeNoNormWEIGHTED-7B-DARETIES](https://huggingface.co/jsfs11/RandomMergeNoNormWEIGHTED-7B-DARETIES) * [senseable/WestLake-7B-v2](https://huggingface.co/senseable/WestLake-7B-v2) * [mlabonne/OmniBeagle-7B](https://huggingface.co/mlabonne/OmniBeagle-7B) * [vanillaOVO/supermario_v3](https://huggingface.co/vanillaOVO/supermario_v3) ## 🧩 Configuration ```yaml base_model: senseable/WestLake-7B-v2 gate_mode: hidden dtype: bfloat16 experts: - source_model: jsfs11/RandomMergeNoNormWEIGHTED-7B-DARETIES positive_prompts: - "Answer this question from the ARC (Argument Reasoning Comprehension)." - "Use common sense and logical reasoning skills." negative_prompts: - "nonsense" - "irrational" - "math" - "code" - source_model: senseable/WestLake-7B-v2 positive_prompts: - "Answer this question from the Winogrande test." - "Use advanced knowledge of culture and humanity" negative_prompts: - "ignorance" - "uninformed" - "creativity" - source_model: mlabonne/OmniBeagle-7B positive_prompts: - "Calculate the answer to this math problem" - "My mathematical capabilities are strong, allowing me to handle complex mathematical queries" - "solve for" negative_prompts: - "incorrect" - "inaccurate" - "creativity" - source_model: vanillaOVO/supermario_v3 positive_prompts: - "Predict the most plausible continuation for this scenario." - "Demonstrate understanding of everyday commonsense in your response." - "Use contextual clues to determine the most likely outcome." - "Apply logical reasoning to complete the given narrative." - "Infer the most realistic action or event that follows." negative_prompts: - "guesswork" - "irrelevant information" - "contradictory response" - "illogical conclusion" - "ignoring context" ``` ## 💻 Usage ```python !pip install -qU transformers bitsandbytes accelerate from transformers import AutoTokenizer import transformers import torch model = "jsfs11/MixtureofMerges-MoE-4x7b-v3" tokenizer = AutoTokenizer.from_pretrained(model) pipeline = transformers.pipeline( "text-generation", model=model, model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True}, ) messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}] prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) print(outputs[0]["generated_text"]) ```