README.md

---
language:
- en
license: gemma
library_name: transformers
tags:
- code
datasets:
- LimYeri/LeetCode_with_Solutions
pipeline_tag: text-generation
model-index:
- name: CodeMind-Gemma-7B-QLoRA-4bit
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 23.21
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 31.17
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 23.12
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 0.0
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 60.3
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 0.0
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
      name: Open LLM Leaderboard
---

# CodeMind
Coding Test Explanatory LLM Model.

## Model Details
- **Model Name**: CodeMind
- **Base Model**: [gemma-7b-it](https://huggingface.co/google/gemma-7b-it)
- **Fine-tuning Datasets**:
  - [LimYeri/LeetCode_with_Solutions](https://huggingface.co/datasets/LimYeri/LeetCode_with_Solutions)
- **Model Type**: Language Model
- **Language**: English
- **License**: gemma
- **Model Size**: 8.54B

- Developed by: [Lim Yeri]
- Contact: [yeari0122@naver.com]

## Intended Use
CodeMind is a fine-tuned language model specifically designed to assist users with coding test questions and provide programming education. It leverages the knowledge from LeetCode user solutions and YouTube video captions related to LeetCode problems to offer guidance, explanations, and code examples.

## Training Data
The model was fine-tuned using the following datasets:
1. **LimYeri/LeetCode_with_Solutions**: This dataset contains Leetcode problems along with their hints, user solutions that have received at least 10 votes, and summaries of Leetcode solution videos from YouTube. These summaries have been processed using the Chain of Thought (CoT) method via commercial Large Language Model (LLM). The 'content' column houses the solutions and captions(CoT Summary), providing detailed explanations, thought processes, and step-by-step instructions for solving the coding problems.

## Training Procedure
- The model was fine-tuned using the Hugging Face Transformer library. The base model, [gemma-7b-it](https://huggingface.co/google/gemma-7b-it), was further trained on the combined dataset of LeetCode user solutions and YouTube video captions(CoT Summary).  This fine-tuning process was designed to enhance the model's understanding of coding concepts and problem-solving strategies, and improve its ability to generate relevant code snippets and explanations.
- The model was trained using the QLoRA technique with 4-bit quantization on the dataset.

## Usage
To use the CodeMind model, you can access it through the Hugging Face model hub or by integrating it into your own applications using the provided API. Provide a coding problem or a question related to programming concepts, and the model will generate relevant explanations, code snippets, or guidance based on its training.

Please refer to the documentation and examples for detailed instructions on how to integrate and use the CodeMind model effectively.

Below we share some code snippets on how to get quickly started with running the model. After downloading the transformers library via 'pip install -U transformers', use the following snippet code.
#### Running the model on a CPU

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")
tokenizer = AutoTokenizer.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")

def get_completion(query: str, model, tokenizer) -> str:
  prompt_template = """
  <start_of_turn>user
  Below is an instruction that describes a task. Write a response that appropriately completes the request.
  {query}
  <end_of_turn>\n\n<start_of_turn>model

  """
  prompt = prompt_template.format(query=query)
  encodeds = tokenizer(prompt, return_tensors="pt", add_special_tokens=True)
  generated_ids = model.generate(**encodeds, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)
  # decoded = tokenizer.batch_decode(generated_ids)
  decoded = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
  return (decoded)

result = get_completion(query="Tell me how to solve the Leetcode Two Sum problem", model=model, tokenizer=tokenizer)
print(result)

```

#### Running the model on a single / multi GPU

```python
# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")
tokenizer = AutoTokenizer.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")

def get_completion(query: str, model, tokenizer) -> str:
  device = "cuda:0"
  prompt_template = """
  <start_of_turn>user
  Below is an instruction that describes a task. Write a response that appropriately completes the request.
  {query}
  <end_of_turn>\n\n<start_of_turn>model

  """
  prompt = prompt_template.format(query=query)
  encodeds = tokenizer(prompt, return_tensors="pt", add_special_tokens=True)
  model_inputs = encodeds.to(device)
  generated_ids = model.generate(**model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)
  # decoded = tokenizer.batch_decode(generated_ids)
  decoded = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
  return (decoded)

result = get_completion(query="Tell me how to solve the Leetcode Two Sum problem", model=model, tokenizer=tokenizer)
print(result)
```

## Bias and Limitations
- The model's knowledge is primarily based on the LeetCode user solutions and YouTube video captions(CoT Summary) used for fine-tuning. It may have limitations in handling coding problems or concepts that are not well-represented in the training data.
- The model's responses are generated based on patterns and information learned from the training data. It may sometimes produce incorrect or suboptimal solutions. Users should always review and verify the generated code before using it in practice.
- The model may exhibit biases present in the training data, such as favoring certain programming styles, algorithms, or approaches. It is important to consider alternative solutions and best practices when using the model's outputs.

## Ethical Considerations
- The model should be used as a supportive tool for learning and problem-solving, not as a substitute for human expertise and critical thinking.
- Users should be aware that the model's responses are generated based on patterns in the training data and may not always be accurate, complete, or up to date.
- The model should not be relied upon for making critical decisions or solving real-world problems without thorough validation and testing.
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_LimYeri__CodeMind-Gemma-7B-QLoRA-4bit)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |22.97|
|AI2 Reasoning Challenge (25-Shot)|23.21|
|HellaSwag (10-Shot)              |31.17|
|MMLU (5-Shot)                    |23.12|
|TruthfulQA (0-shot)              | 0.00|
|Winogrande (5-shot)              |60.30|
|GSM8k (5-shot)                   | 0.00|