README.md · nazimali/Mistral-Nemo-Kurdish-Instruct at 027c37e5193a54c14e638c7959530d3a9e8fdae4

metadata

base_model:
  - nazimali/Mistral-Nemo-Kurdish
language:
  - ku
  - en
license: apache-2.0
tags:
  - text-generation-inference
  - transformers
  - unsloth
  - mistral
  - gguf
datasets:
  - saillab/alpaca-kurdish_kurmanji-cleaned
library_name: transformers

This is a 12B parameter model, finetuned on nazimali/Mistral-Nemo-Kurdish for a single Kurdish (Kurmanji) instruction dataset. My intention was to train this with both Kurdish Kurmanji Latin script and Kurdish Sorani Arabic script, but training time was much longer than anticipated. So I decided to use 1 full Kurdish Kurmanji dataset to get started.

Will look into a multi-GPU training setup so don't have to wait all day for results. Want to train it with both Kurmanji and Sorani Arabic script.

Try spaces demo example.

Example usage

llama-cpp-python

from llama_cpp import Llama

inference_prompt = """Li jêr rêwerzek heye ku peywirek rave dike, bi têketinek ku çarçoveyek din peyda dike ve tê hev kirin. Bersivek ku daxwazê bi guncan temam dike binivîsin.
### Telîmat:
{}
### Têketin:
{}
### Bersiv:
"""

llm = Llama.from_pretrained(
    repo_id="nazimali/Mistral-Nemo-Kurdish-Instruct",
    filename="Q4_K_M.gguf",
)

llm.create_chat_completion(
    messages = [
        {
            "role": "user",
            "content": inference_prompt.format("selam alikum, tu çawa yî?")
        }
    ]
)

llama.cpp

./llama-cli \
  --hf-repo "nazimali/Mistral-Nemo-Kurdish-Instruct" \
  --hf-file Q4_K_M.gguf \
  -p "selam alikum, tu çawa yî?" \
  --conversation

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "nazimali/Mistral-Nemo-Kurdish-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

Training

Transformers 4.44.2
1 NVIDIA A40
Duration 7h 41m 12s

{
  "total_flos": 2225817933447045000,
  "train/epoch": 0.9998075072184792,
  "train/global_step": 2597,
  "train/grad_norm": 1.172538161277771,
  "train/learning_rate": 0,
  "train/loss": 0.7774,
  "train_loss": 0.892096030377038,
  "train_runtime": 27479.3172,
  "train_samples_per_second": 1.512,
  "train_steps_per_second": 0.095
}

Finetuning data:

saillab/alpaca-kurdish_kurmanji-cleaned
Dataset number of rows: 52,002
Filtered columns instruction, output
- Must have at least 1 character
- Must be less than 10,000 characters
Number of rows used for training: 41,559

Finetuning instruction format:

finetune_prompt = """Li jêr rêwerzek heye ku peywirek rave dike, bi têketinek ku çarçoveyek din peyda dike ve tê hev kirin. Bersivek ku daxwazê bi guncan temam dike binivîsin.
### Telîmat:
{}
### Têketin:
{}
### Bersiv:
{}
"""