metadata

license: llama3
language:
  - tr
model-index:
  - name: Kocdigital-LLM-8b-v0.1
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge TR
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc
            value: 44.03
            name: accuracy
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag TR
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc
            value: 46.73
            name: accuracy
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU TR
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 49.11
            name: accuracy
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA TR
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: acc
            name: accuracy
            value: 48.21
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande TR
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc
            value: 54.98
            name: accuracy
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k TR
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 51.78
            name: accuracy

Kocdigital-LLM-8b-v0.1

This model is an fine-tuned version of a Llama3 8b Large Language Model (LLM) for Turkish. It was trained on a high quality Turkish instruction sets created from various open-source and internal resources. Turkish Instruction dataset carefully annotated to carry out Turkish instructions in an accurate and organized manner. The training process involved using the QLORA method.

Model Details

Base Model: Llama3 8B based LLM
Training Dataset: High Quality Turkish instruction sets
Training Method: SFT with QLORA

QLORA Fine-Tuning Configuration

lora_alpha: 128
lora_dropout: 0
r: 64
target_modules: "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"
bias: "none"

Usage Examples


from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"KOCDIGITAL/Kocdigital-LLM-8b-v0.1", 
max_seq_length=4096)
model = AutoModelForCausalLM.from_pretrained(
    "KOCDIGITAL/Kocdigital-LLM-8b-v0.1",
    load_in_4bit=True,
)

system = 'Sen Türkçe konuşan genel amaçlı bir asistansın. Her zaman kullanıcının verdiği talimatları doğru, kısa ve güzel bir gramer ile yerine getir.'

template = "{}\n\n###Talimat\n{}\n###Yanıt\n"
content = template.format(system, 'Türkiyenin 3 büyük ilini listeler misin.')

conv = []
conv.append({'role': 'user', 'content': content})
inputs = tokenizer.apply_chat_template(conv, 
                                       tokenize=False, 
                                       add_generation_prompt=True, 
                                       return_tensors="pt")

print(inputs)

inputs = tokenizer([inputs], 
                   return_tensors = "pt",
                   add_special_tokens=False).to("cuda")

outputs = model.generate(**inputs, 
                         max_new_tokens = 512, 
                         use_cache = True, 
                         do_sample = True, 
                         top_k = 50, 
                         top_p = 0.60, 
                         temperature = 0.3, 
                         repetition_penalty=1.1)

out_text = tokenizer.batch_decode(outputs)[0]
print(out_text)

[Open LLM Turkish Leaderboard v0.2 Evaluation Results]

Metric	Value
Avg.	49.11
AI2 Reasoning Challenge_tr-v0.2	44.03
HellaSwag_tr-v0.2	46.73
MMLU_tr-v0.2	49.11
TruthfulQA_tr-v0.2	48.51
Winogrande _tr-v0.2	54.98
GSM8k_tr-v0.2	51.78