metadata

license: apache-2.0
datasets:
  - agentlans/crash-course
  - vicgalle/configurable-system-prompt-multitask
base_model:
  - Qwen/Qwen2.5-0.5B-Instruct
model-index:
  - name: Qwen2.5-0.5B-Instruct-CrashCourse-dropout
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: IFEval (0-Shot)
          type: wis-k/instruction-following-eval
          split: train
          args:
            num_few_shot: 0
        metrics:
          - type: inst_level_strict_acc and prompt_level_strict_acc
            value: 29.49
            name: averaged accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=agentlans%2FQwen2.5-0.5B-Instruct-CrashCourse-dropout
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: BBH (3-Shot)
          type: SaylorTwift/bbh
          split: test
          args:
            num_few_shot: 3
        metrics:
          - type: acc_norm
            value: 7.23
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=agentlans%2FQwen2.5-0.5B-Instruct-CrashCourse-dropout
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MATH Lvl 5 (4-Shot)
          type: lighteval/MATH-Hard
          split: test
          args:
            num_few_shot: 4
        metrics:
          - type: exact_match
            value: 0.08
            name: exact match
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=agentlans%2FQwen2.5-0.5B-Instruct-CrashCourse-dropout
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GPQA (0-shot)
          type: Idavidrein/gpqa
          split: train
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 1.79
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=agentlans%2FQwen2.5-0.5B-Instruct-CrashCourse-dropout
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MuSR (0-shot)
          type: TAUR-Lab/MuSR
          args:
            num_few_shot: 0
        metrics:
          - type: acc_norm
            value: 1.11
            name: acc_norm
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=agentlans%2FQwen2.5-0.5B-Instruct-CrashCourse-dropout
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU-PRO (5-shot)
          type: TIGER-Lab/MMLU-Pro
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 6.76
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/?search=agentlans%2FQwen2.5-0.5B-Instruct-CrashCourse-dropout
          name: Open LLM Leaderboard

Qwen2.5-0.5B-Instruct-CrashCourse-dropout

Model Description

This model is a fine-tuned version of Qwen/Qwen2.5-0.5B-Instruct, specifically adapted for enhanced performance on instructional and multitask scenarios. It leverages two datasets: "agentlans/crash-course" and "vicgalle/configurable-system-prompt-multitask" to improve its capabilities in handling diverse tasks and responding to various instruction formats.

Intended Use

This model is designed for:

Answering questions related to crash course materials
Handling configurable system prompts for multitask scenarios
General instruction-following tasks

Training Procedure

The model was fine-tuned on the specified datasets using the Qwen2.5-0.5B-Instruct as the base model. More details on the training process will be added here later.

Limitations

The model's performance may be biased towards the specific domains covered in the training datasets.
As with all language models, it may occasionally produce inaccurate or inconsistent outputs.
The model's knowledge is limited to the information available in its training data and the base model's knowledge cutoff.

Ethical Considerations

Users should be aware that this model, like all AI models, may reflect biases present in its training data. It's crucial to use the model responsibly and to verify important information from authoritative sources.

Additional Information

For more details on the base model, please refer to the Qwen/Qwen2.5-0.5B-Instruct model card. For information about the datasets used in fine-tuning, check the respective dataset cards on the Hugging Face Hub.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here! Summarized results can be found here!

Metric	Value (%)
Average	7.74
IFEval (0-Shot)	29.49
BBH (3-Shot)	7.23
MATH Lvl 5 (4-Shot)	0.08
GPQA (0-shot)	1.79
MuSR (0-shot)	1.11
MMLU-PRO (5-shot)	6.76