README.md

metadata

base_model:
  - Qwen/Qwen2-1.5B
  - Replete-AI/Replete-Coder-Qwen2-1.5b
license: apache-2.0
tags:
  - moe
  - frankenmoe
  - merge
  - mergekit
  - lazymergekit
  - Qwen/Qwen2-1.5B
  - Replete-AI/Replete-Coder-Qwen2-1.5b

QwenMoEAriel

QwenMoEAriel is a Mixture of Experts (MoE) made with the following models using LazyMergekit:

• Qwen/Qwen2-1.5B
• Replete-AI/Replete-Coder-Qwen2-1.5b

🧩 Configuration

base_model: Qwen/Qwen2-1.5B
architecture: qwen
experts:
  - source_model: Qwen/Qwen2-1.5B
    positive_prompts:
    - "chat"
    - "assistant"
    - "tell me"
    - "explain"
    - "I want"
  - source_model: Replete-AI/Replete-Coder-Qwen2-1.5b
    positive_prompts:
    - "code"
    - "python"
    - "javascript"
    - "programming"
    - "algorithm"
shared_experts:
  - source_model: Qwen/Qwen2-1.5B
    positive_prompts: # required by Qwen MoE for "hidden" gate mode, otherwise not allowed
      - "chat"
    # (optional, but recommended:)
    residual_scale: 0.1 # downweight output from shared expert to prevent overcooking the model

💻 Usage

!pip install -qU transformers bitsandbytes accelerate einops
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)
model = AutoModelForCausalLM.from_pretrained(
    "femiari/Qwen2-1.5Moe",
    torch_dtype=torch.float16,
    ignore_mismatched_sizes=True
).to(device)
tokenizer = AutoTokenizer.from_pretrained("femiari/Qwen2-1.5Moe")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

print(response)