leaderboard-pr-bot's picture
Adding Evaluation Results
fd333ff verified
|
raw
history blame
6.82 kB
metadata
license: llama3
library_name: transformers
tags:
  - mergekit
  - merge
base_model:
  - Undi95/Meta-Llama-3-8B-Instruct-hf
model-index:
  - name: Llama-3-8B-Ultra-Instruct
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 64.59
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 81.63
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 68.32
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 52.8
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 76.95
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 70.36
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=elinas/Llama-3-8B-Ultra-Instruct
          name: Open LLM Leaderboard

Llama-3-8B-Ultra-Instruct

This is a merge of pre-trained language models created using mergekit.

Hello everyone this is Dampf, creator of the Destroyer series!

looks around Oh, now I'm on Elinas' HF account. As you can see, I'm quite the traveler!

This time, I'm introducing you to 8B-Ultra-Instruct. It is a small general purpose model that combines the most powerful instruct models with enticing roleplaying models. It will introduce better RAG capabilities in the form of Bagel to Llama 3 8B Instruct as well as German multilanguage, higher general intelligence and vision support. A model focused on Biology adds knowledge in the medical field.

As for roleplay, it features two of the hottest models right now. Those are known to be high quality and for being uncensored. So this model might put out harmful responses. We are not responsible for what you do with this model and please take everything the model says with a huge grain of salt. Lastly, you might notice I'm conversative with the weight values in the final merge. This is because I believe L8B Instruct is a very dense model that's already great and doesn't need a lot more data. So instead of reaching the weight value of 1 in a ties merge, I'm only using a total of 0,65. This is to preserve Llama Instruct's intelligence and knowledge, while adding a little bit of the aforementioned models as salt in the soup.

A huge thank you for all the creators of the datasets. Those include Undi95, Jon Durbin, Aaditya, VAGOsolutions, Teknium, Camel and many more. They deserve all the credit. And of course, thank you Elinas for providing the compute.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using Undi95/Meta-Llama-3-8B-Instruct-hf as a base.

Models Merged

The following models were included in the merge:

  • llama-3-8B-ultra-instruct/InstructPart
  • llama-3-8B-ultra-instruct/RPPart

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: ChaoticNeutrals/Poppy_Porpoise-v0.7-L3-8B
    parameters:
      weight: 0.4
  - model: Undi95/Llama-3-LewdPlay-8B-evo
    parameters:
      weight: 0.5
  - model: jondurbin/bagel-8b-v1.0
    parameters:
      weight: 0.1
merge_method: dare_ties
dtype: bfloat16
base_model: Undi95/Meta-Llama-3-8B-hf
name: RPPart
---
models:
  - model: Weyaxi/Einstein-v6.1-Llama3-8B
    parameters:
      weight: 0.6
  - model: VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct
    parameters:
      weight: 0.3
  - model: aaditya/OpenBioLLM-Llama3-8B
    parameters:
      weight: 0.1
merge_method: dare_ties
base_model: Undi95/Meta-Llama-3-8B-hf
dtype: bfloat16
name: InstructPart
---
models:
  - model: RPPart
    parameters:
      weight: 0.39
  - model: InstructPart
    parameters:
      weight: 0.26
merge_method: dare_ties
base_model: Undi95/Meta-Llama-3-8B-Instruct-hf
dtype: bfloat16
name: Llama-3-8B-Ultra-Instruct 

Chat Template (Llama 3 Official)

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{input}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{output}<|eot_id|>

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 69.11
AI2 Reasoning Challenge (25-Shot) 64.59
HellaSwag (10-Shot) 81.63
MMLU (5-Shot) 68.32
TruthfulQA (0-shot) 52.80
Winogrande (5-shot) 76.95
GSM8k (5-shot) 70.36