sinhala-llama-2-7b-chat-hf

This model is a fine-tuned version of NousResearch/Llama-2-7b-chat-hf on the Thimira/sinhala-llm-dataset-llama-prompt-format dataset.

Model description

This is a model for Sinhala language text generation which is fine-tuned from the base llama-2-7b-chat-hf model.

Currently the capabilities of themodel are extremely limited, and requires further data and fine-tuning to be useful. Feel free to experiment with the model and provide feedback.

Usage example

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

tokenizer = AutoTokenizer.from_pretrained("Thimira/sinhala-llama-2-7b-chat-hf")
model = AutoModelForCausalLM.from_pretrained("Thimira/sinhala-llama-2-7b-chat-hf")

pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=200)

prompt = "ඔබට සිංහල භාෂාව තේරුම් ගත හැකිද?"
result = pipe(f"<s>[INST] {prompt} [/INST]")
print(result[0]['generated_text'])

Intended uses & limitations

The Sinhala-LLaMA models are intended for assistant-like chat in the Sinhala language.

To get the expected features and performance from these models the LLaMA 2 prompt format needs to be followed, including the INST and <> tags, BOS and EOS tokens, and the whitespaces and breaklines in between.

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 2
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.03
  • num_epochs: 2

Training results

Framework versions

  • PEFT 0.10.0
  • Transformers 4.40.2
  • Pytorch 2.1.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
790
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for Thimira/sinhala-llama-2-7b-chat-hf

Adapter
(367)
this model

Dataset used to train Thimira/sinhala-llama-2-7b-chat-hf