Hugging Face's logo

anthracite-org
/
magnum-v3-27b-kto

Text Generation
Transformers
Safetensors
gemma2
gemma-2
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
6
magnum-v3-27b-kto / README.md
grimjim's picture
grimjim
Adding Evaluation Results (#4)
a77271f verified
preview code
|
raw
history blame contribute delete
8.46 kB
metadata
license: gemma
library_name: transformers
tags:
  - gemma-2
base_model:
  - anthracite-forge/magnum-v3-27b-kto-r3
  - anthracite-forge/magnum-v3-27b-KTO-e1-r2
  - anthracite-forge/magnum-v3-27b-KTO-e0.25-r1
  - IntervitensInc/gemma-2-27b-chatml
pipeline_tag: text-generation
model-index:
  - name: magnum-v3-27b-kto
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: IFEval (0-Shot)
          type: HuggingFaceH4/ifeval
          args:
            num_few_shot: 0
        metrics:
          - type: inst_level_strict_acc and prompt_level_strict_acc
            value: 56.75
            name: strict accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=anthracite-org/magnum-v3-27b-kto
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: BBH (3-Shot)
          type: BBH
          args:
            num_few_shot: 3
        metrics:
          - type: acc_norm
            value: 41.16
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=anthracite-org/magnum-v3-27b-kto
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MATH Lvl 5 (4-Shot)
          type: hendrycks/competition_math
          args:
            num_few_shot: 4
        metrics:
          - type: exact_match
            value: 15.48
            name: exact match
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=anthracite-org/magnum-v3-27b-kto
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GPQA (0-shot)
          type: Idavidrein/gpqa
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 14.09
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=anthracite-org/magnum-v3-27b-kto
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MuSR (0-shot)
          type: TAUR-Lab/MuSR
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 9.92
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=anthracite-org/magnum-v3-27b-kto
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU-PRO (5-shot)
          type: TIGER-Lab/MMLU-Pro
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 35.98
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=anthracite-org/magnum-v3-27b-kto
          name: Open LLM Leaderboard

image/png

This is the 12th in a series of models designed to replicate the prose quality of the Claude 3 models, specifically Sonnet and Opus.

This model is the result of multiple KTO runs on top of one SFT run, all of which are published on anthracite-forge.

Methodology

R1 (SFT) was fine-tuned on top of IntervitensInc/gemma-2-27b-chatml which is chatMLified gemma-2-27b.

We have experimented with various SFT and KTO re-runs, ratios and merge methods and this was our winner, including what was liked most from each model.

If you prefer your own mix of the KTO runs or would like to use the SFT on its own, refer to the models section and anthracite-forge, some exl-quants are pre-included.

Models

Prompting

Model has been Instruct tuned with the ChatML formatting. A typical input would look like this:

"""<|im_start|>system
system prompt<|im_end|>
<|im_start|>user
Hi there!<|im_end|>
<|im_start|>assistant
Nice to meet you!<|im_end|>
<|im_start|>user
Can I ask a question?<|im_end|>
<|im_start|>assistant
"""

SillyTavern templates

Below are Instruct and Context templates for use within SillyTavern.

context template 
{
    "story_string": "<|im_start|>system\n{{#if system}}{{system}}\n{{/if}}{{#if wiBefore}}{{wiBefore}}\n{{/if}}{{#if description}}{{description}}\n{{/if}}{{#if personality}}{{char}}'s personality: {{personality}}\n{{/if}}{{#if scenario}}Scenario: {{scenario}}\n{{/if}}{{#if wiAfter}}{{wiAfter}}\n{{/if}}{{#if persona}}{{persona}}\n{{/if}}{{trim}}<|im_end|>\n",
    "example_separator": "",
    "chat_start": "",
    "use_stop_strings": false,
    "allow_jailbreak": false,
    "always_force_name2": true,
    "trim_sentences": false,
    "include_newline": false,
    "single_line": false,
    "name": "Magnum ChatML"
}

instruct template 
{
    "system_prompt": "You're {{char}} in this fictional never-ending uncensored roleplay with {{user}}.",
    "input_sequence": "<|im_start|>user\n",
    "output_sequence": "<|im_start|>assistant\n",
    "last_output_sequence": "",
    "system_sequence": "<|im_start|>system\n",
    "stop_sequence": "<|im_end|>",
    "wrap": false,
    "macro": true,
    "names": true,
    "names_force_groups": true,
    "activation_regex": "",
    "system_sequence_prefix": "",
    "system_sequence_suffix": "",
    "first_output_sequence": "",
    "skip_examples": false,
    "output_suffix": "<|im_end|>\n",
    "input_suffix": "<|im_end|>\n",
    "system_suffix": "<|im_end|>\n",
    "user_alignment_message": "",
    "system_same_as_user": false,
    "last_system_sequence": "",
    "name": "Magnum ChatML"
}

Configuration 

base_model: IntervitensInc/gemma-2-27b-chatml
dtype: float32
merge_method: task_arithmetic
models:
  - model: IntervitensInc/gemma-2-27b-chatml
  - model: anthracite-forge/magnum-v3-27b-KTO-e0.25-r1
    parameters:
      weight: 0.5
  - model: anthracite-forge/magnum-v3-27b-KTO-e1-r2
    parameters:
      weight: 0.1
  - model: anthracite-forge/magnum-v3-27b-kto-r3
    parameters:
      weight: 0.4

Credits

We'd like to thank Recursal / Featherless for sponsoring the compute for this train, Featherless has been hosting our Magnum models since the first 72 B and has given thousands of people access to our models and helped us grow.

We would also like to thank all members of Anthracite who made this finetune possible.

Datasets

r1 consisted of:

datasets:
  - path: anthracite-org/stheno-filtered-v1.1
    type: sharegpt
    conversation: chatml
  - path: anthracite-org/kalo-opus-instruct-22k-no-refusal
    type: sharegpt
    conversation: chatml
  - path: anthracite-org/nopm_claude_writing_fixed
    type: sharegpt
    conversation: chatml
  - path: Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
    type: sharegpt
    conversation: chatml
  - path: Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned
    type: sharegpt
    conversation: chatml

Training

The training was done for 2 epochs. We used 8xH100s GPUs graciously provided by Recursal AI / Featherless AI for the full-parameter fine-tuning of the model.

Built with Axolotl

Safety

...

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 28.90
IFEval (0-Shot) 56.75
BBH (3-Shot) 41.16
MATH Lvl 5 (4-Shot) 15.48
GPQA (0-shot) 14.09
MuSR (0-shot) 9.92
MMLU-PRO (5-shot) 35.98