README.md

metadata

language:
  - en
license: gemma
library_name: transformers
tags:
  - code
datasets:
  - LimYeri/LeetCode_with_Solutions
pipeline_tag: text-generation
model-index:
  - name: CodeMind-Gemma-7B-QLoRA-4bit
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 23.21
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 31.17
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 23.12
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 0
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 60.3
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 0
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=LimYeri/CodeMind-Gemma-7B-QLoRA-4bit
          name: Open LLM Leaderboard

CodeMind

Coding Test Explanatory LLM Model.

Model Details

• Model Name: CodeMind

• Base Model: gemma-7b-it

• Fine-tuning Datasets:

  • LimYeri/LeetCode_with_Solutions

• Model Type: Language Model

• Language: English

• License: gemma

• Model Size: 8.54B

• Developed by: [Lim Yeri]

• Contact: [yeari0122@naver.com]

Intended Use

CodeMind is a fine-tuned language model specifically designed to assist users with coding test questions and provide programming education. It leverages the knowledge from LeetCode user solutions and YouTube video captions related to LeetCode problems to offer guidance, explanations, and code examples.

Training Data

The model was fine-tuned using the following datasets:

1. LimYeri/LeetCode_with_Solutions: This dataset contains Leetcode problems along with their hints, user solutions that have received at least 10 votes, and summaries of Leetcode solution videos from YouTube. These summaries have been processed using the Chain of Thought (CoT) method via commercial Large Language Model (LLM). The 'content' column houses the solutions and captions(CoT Summary), providing detailed explanations, thought processes, and step-by-step instructions for solving the coding problems.

Training Procedure

• The model was fine-tuned using the Hugging Face Transformer library. The base model, gemma-7b-it, was further trained on the combined dataset of LeetCode user solutions and YouTube video captions(CoT Summary). This fine-tuning process was designed to enhance the model's understanding of coding concepts and problem-solving strategies, and improve its ability to generate relevant code snippets and explanations.
• The model was trained using the QLoRA technique with 4-bit quantization on the dataset.

Usage

To use the CodeMind model, you can access it through the Hugging Face model hub or by integrating it into your own applications using the provided API. Provide a coding problem or a question related to programming concepts, and the model will generate relevant explanations, code snippets, or guidance based on its training.

Please refer to the documentation and examples for detailed instructions on how to integrate and use the CodeMind model effectively.

Below we share some code snippets on how to get quickly started with running the model. After downloading the transformers library via 'pip install -U transformers', use the following snippet code.

Running the model on a CPU

from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")
tokenizer = AutoTokenizer.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")

def get_completion(query: str, model, tokenizer) -> str:
  prompt_template = """
  <start_of_turn>user
  Below is an instruction that describes a task. Write a response that appropriately completes the request.
  {query}
  <end_of_turn>\n\n<start_of_turn>model

  """
  prompt = prompt_template.format(query=query)
  encodeds = tokenizer(prompt, return_tensors="pt", add_special_tokens=True)
  generated_ids = model.generate(**encodeds, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)
  # decoded = tokenizer.batch_decode(generated_ids)
  decoded = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
  return (decoded)

result = get_completion(query="Tell me how to solve the Leetcode Two Sum problem", model=model, tokenizer=tokenizer)
print(result)

Running the model on a single / multi GPU

# pip install accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")
tokenizer = AutoTokenizer.from_pretrained("LimYeri/CodeMind-Gemma-7B-QLoRA-4bit")

def get_completion(query: str, model, tokenizer) -> str:
  device = "cuda:0"
  prompt_template = """
  <start_of_turn>user
  Below is an instruction that describes a task. Write a response that appropriately completes the request.
  {query}
  <end_of_turn>\n\n<start_of_turn>model

  """
  prompt = prompt_template.format(query=query)
  encodeds = tokenizer(prompt, return_tensors="pt", add_special_tokens=True)
  model_inputs = encodeds.to(device)
  generated_ids = model.generate(**model_inputs, max_new_tokens=1000, do_sample=True, pad_token_id=tokenizer.eos_token_id)
  # decoded = tokenizer.batch_decode(generated_ids)
  decoded = tokenizer.decode(generated_ids[0], skip_special_tokens=True)
  return (decoded)

result = get_completion(query="Tell me how to solve the Leetcode Two Sum problem", model=model, tokenizer=tokenizer)
print(result)

Bias and Limitations

• The model's knowledge is primarily based on the LeetCode user solutions and YouTube video captions(CoT Summary) used for fine-tuning. It may have limitations in handling coding problems or concepts that are not well-represented in the training data.
• The model's responses are generated based on patterns and information learned from the training data. It may sometimes produce incorrect or suboptimal solutions. Users should always review and verify the generated code before using it in practice.
• The model may exhibit biases present in the training data, such as favoring certain programming styles, algorithms, or approaches. It is important to consider alternative solutions and best practices when using the model's outputs.

Ethical Considerations

• The model should be used as a supportive tool for learning and problem-solving, not as a substitute for human expertise and critical thinking.
• Users should be aware that the model's responses are generated based on patterns in the training data and may not always be accurate, complete, or up to date.
• The model should not be relied upon for making critical decisions or solving real-world problems without thorough validation and testing.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	22.97
AI2 Reasoning Challenge (25-Shot)	23.21
HellaSwag (10-Shot)	31.17
MMLU (5-Shot)	23.12
TruthfulQA (0-shot)	0.00
Winogrande (5-shot)	60.30
GSM8k (5-shot)	0.00