autocodit / README.md
adowu's picture
Adding Evaluation Results
afd6d45 verified
|
raw
history blame
5.85 kB
---
language:
- en
license: apache-2.0
library_name: transformers
base_model:
- HuggingFaceH4/mistral-7b-anthropic
- ajibawa-2023/Code-Mistral-7B
- Undi95/BigL-7B
model-index:
- name: autocodit
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 66.38
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 84.82
name: normalized accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 65.09
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 59.95
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 80.51
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 60.65
name: accuracy
source:
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=adowu/autocodit
name: Open LLM Leaderboard
---
# AUTOCODIT
# Description
This model represents an innovative fusion of three cutting-edge language models: **BigL-7B**, **Code-Mistral-7B**, and **mistral-7b-anthropic**, leveraging the strengths of each to create a more powerful and versatile tool. The integration process employs the TIES merge method, meticulously combining these models to enhance performance and adaptability across a broad spectrum of natural language processing tasks.
**Creation Process**
The model was crafted through a strategic merging process, utilizing the TIES merge method. This approach was chosen for its effectiveness in preserving the unique capabilities of each constituent model while ensuring seamless interoperability. The base model for this fusion was **HuggingFaceH4/mistral-7b-anthropic**, selected for its robust architecture and performance.
The merge parameters were carefully calibrated to achieve the optimal balance between the models, with the following configuration:
- **BigL-7B** was integrated with a density of 0.9 and a weight of 0.8, contributing its extensive language understanding and generation capabilities.
- **Code-Mistral-7B** was incorporated with a density of 0.7 and a weight of 0.7, enhancing the model's proficiency in code-related tasks and technical language comprehension.
- **mistral-7b-anthropic** served as the foundation, with its parameters set to a density of 0.9 and a weight of 0.8, ensuring the model's general language processing abilities remained at the forefront.
**Features**
- **Model Type:** `MistralForCausalLM`
- **Vocabulary Size:** 32,000 tokens, encompassing a wide array of linguistic elements for comprehensive language coverage.
- **Maximum Position Embeddings:** 32,768, facilitating the processing of extended passages of text.
- **Hidden Size:** 4,096, enabling the model to capture complex patterns and nuances in the data.
- **Num Attention Heads:** 32, allowing for detailed attention to various aspects of the input.
- **Num Hidden Layers:** 32, providing depth to the model's understanding and generation capabilities.
**Applications**
This model is adept at a wide range of natural language processing tasks, including but not limited to text generation, language translation, code synthesis, and more. Its unique blend of features from **BigL-7B**, **Code-Mistral-7B**, and **mistral-7b-anthropic** makes it particularly effective in scenarios requiring a deep understanding of both human and programming languages.
---
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_adowu__autocodit)
| Metric |Value|
|---------------------------------|----:|
|Avg. |69.57|
|AI2 Reasoning Challenge (25-Shot)|66.38|
|HellaSwag (10-Shot) |84.82|
|MMLU (5-Shot) |65.09|
|TruthfulQA (0-shot) |59.95|
|Winogrande (5-shot) |80.51|
|GSM8k (5-shot) |60.65|