Edit model card

Just 11B RP experemental merge. For some time I wanted to do something similar.

Model used

Prompt template

Just use alpaca or chatml. Both works fine.

The secret sauce

lemolake-11B :

slices:
  - sources:
    - model: KatyTheCutie/LemonadeRP-4.5.3
      layer_range: [0, 24]
  - sources:
    - model: senseable/WestLake-7B-v2
      layer_range: [8, 32]

merge_method: passthrough
dtype: bfloat16

holobus-11B :

slices:
  - sources:
    - model: KoboldAI/Mistral-7B-Holodeck-1
      layer_range: [0, 24]
  - sources:
    - model: KoboldAI/Mistral-7B-Erebus-v3
      layer_range: [8, 32]

merge_method: passthrough
dtype: bfloat16

Vertilake-11B :

base_model: "Mistral-11B-v0.1"
models:
  - model: "Mistral-11B-v0.1"
    # no parameters necessary for base model
  - model: "Fimbulvetr-11B-v2"
    parameters:
      weight: 0.43
      density: 0.8
  - model: "lemolake-11b"
    parameters:
      weight: 0.6
      density: 0.8
  - model: "Holobus-11B"
    parameters:
      weight: 0.17
      density: 0.5
merge_method: dare_ties
parameters:
  int8_mask: true
dtype: bfloat16

I use mergekit for all the manipulation told here.

Thanks to the Undi95 for the original 11B mistral merge recipe.

Downloads last month
16
Safetensors
Model size
10.7B params
Tensor type
BF16
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for beberik/Vertilake-11B

Quantizations
2 models