FMixIA-FrankenMerge-9.5B-PT-9

A merged model using Passthrough layer concatenation, creating a frankenmerge model using mergekit.

Model Details

• Base Models:
  • ZeroXClem/Qwen2.5-7B-HomerCreative-Mix
  • ZeroXClem/Qwen2.5-7B-HomerAnvita-NerdMix
• Merge Method: passthrough
• Note: This is a frankenmerge model with modified architecture

Configuration

slices:
  - sources:
    - model: ZeroXClem/Qwen2.5-7B-HomerCreative-Mix
      layer_range: [0, 28]
  - sources:
    - model: ZeroXClem/Qwen2.5-7B-HomerAnvita-NerdMix
      layer_range: [0, 28]
merge_method: passthrough
dtype: bfloat16

Usage

This model can be used with the standard transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Ro-xe/FMixIA-FrankenMerge-9.5B-PT-9")
tokenizer = AutoTokenizer.from_pretrained("Ro-xe/FMixIA-FrankenMerge-9.5B-PT-9")