---
library_name: transformers
license: apache-2.0
datasets:
- yusufbaykaloglu/University_Mevzuat_QA_v2
language:
- tr
base_model:
- ytu-ce-cosmos/Turkish-Llama-8b-DPO-v0.1
pipeline_tag: text-generation
tags:
- legal
---

## Model Details

### Model Description

his model is based on the **ytu-ce-cosmos/Turkish-Llama-8b-DPO-v0.1** base model and has been fine-tuned on a Turkish Q&A dataset concerning university regulations. The fine-tuning was performed using the LoRA method with 4-bit quantization via BitsAndBytes to reduce memory footprint while preserving performance. The training data, sourced from the dataset [yusufbaykaloglu/University_Mevzuat_QA_v2](https://huggingface.co/datasets/yusufbaykaloglu/University_Mevzuat_QA_v2), contains question–answer pairs formatted as follows:

```python
<s>[INST] {question} [/INST] {answer}</s>
```

- **Developed by:** [Yusuf Baykaloğlu]
- **Model type:** [Causal Language Model fine-tuned with LoRA]
- **Language(s) (NLP):** [Turkish]
- **Finetuned from model:** [ytu-ce-cosmos/Turkish-Llama-8b-DPO-v0.1]


## How to Get Started with the Model

Below is a sample code snippet to get started with the model:

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "yusufbaykaloglu/Turkish-University-Regulations"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_name)
model.to(device)

input_text = "<s>[INST] Sorunuzu giriniz. [/INST]<s>"
inputs = tokenizer(input_text, return_tensors="pt").to(device)
outputs = model.generate(inputs["input_ids"], max_length=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```
### Example

```python
input_text = "<s>[INST] Dokuz Eylül Üniversitesi Fen Fakültesi öğrencisiyim. İngilizce hazırlık programından muaf olmak veya doğrudan atlamak için hangi şartları sağlamam gerekiyor?  [/INST]<s>"

<s>[INST] Dokuz Eylül Üniversitesi Fen Fakültesi öğrencisiyim. İngilizce hazırlık programından muaf olmak veya doğrudan atlamak için hangi şartları sağlamam gerekiyor?

[/INST]<s>Madde 6'ya göre, İngilizce hazırlık programından muaf olmak veya doğrudan atlamak için aşağıdaki şartları sağlamanız gerekiyor:
  1. **Yabancı Dil Yeterlik Sınavı:** Yabancı dil yeterliliğinizi kanıtlamak için Yabancı Dil Yeterlik Sınavı'na girmelisiniz.
  2. **Yeterlik Notu:** Sınavda belirli bir not almanız gerekiyor. Bu not, İngilizce hazırlık programından muafiyet veya atlamayı sağlayacak düzeyde belirlenmiştir.
  3. **Muafiyet veya Atlamayı Başvurma:** Sınav sonuçlarınızı ve gerekli belgeleri, fakültenin belirlediği prosedürlere uygun olarak Öğrenci İşleri Daire Başkanlığına sunmanız gerekir.
  4. **Onay:** Öğrenci İşleri Daire Başkanlığı, başvurunuzu inceleyecek ve muafiyet veya atlamayı onaylayacaktır. 
```

## Bias, Risks, and Limitations

This model has been trained on a specific Q&A dataset related to Turkish university regulations and may reflect inherent biases or gaps present in the source data. Users should be aware that:

- Generated responses might occasionally be incorrect, incomplete, or outdated.
- The model is best used as an assistive tool rather than a sole authority on university regulations.



## Training Details

### Training Data

- **Source:** The model was fine-tuned using the dataset [yusufbaykaloglu/University_Mevzuat_QA_v2](https://huggingface.co/datasets/yusufbaykaloglu/University_Mevzuat_QA_v2), which comprises Turkish question–answer pairs on university regulations.


### Training Procedure

- **Base Model:**  
  - `ytu-ce-cosmos/Turkish-Llama-8b-DPO-v0.1`

- **Fine-Tuning Method:**  
  - LoRA (Low-Rank Adaptation)

- **LoRA Configuration:**  
  - `r = 64`  
  - `lora_alpha = 128`  
  - `lora_dropout = 0.1`  
  - `bias = "none"`  
  - Task type: `CAUSAL_LM`

- **Training Framework:**  
  - SFTTrainer from the TRL library

- **Training Hyperparameters:**  
  - **Epochs:** 8  
  - **Per-device Batch Size:** 2  
  - **Gradient Accumulation Steps:** 16  
  - **Learning Rate:** 1e-4  
  - **Weight Decay:** 0.001  
  - **Warmup Ratio:** 0.03 (with 100 warmup steps)  
  - **LR Scheduler:** Cosine  
  - **Max Sequence Length:** 256 tokens  
  - **Optimizer:** paged_adamw_32bit  
  - **Max Gradient Norm:** 0.3

- **Quantization:**  
  - 4-bit quantization using BitsAndBytes for efficient memory and compute usage


### Training Loss

Below are the training loss values recorded at various training steps:

| Step | Training Loss |
|------|---------------|
| 500  | 1.077800      |
| 1000 | 0.866600      |
| 1500 | 0.791500      |
| 2000 | 0.736100      |
| 2500 | 0.693900      |
| 3000 | 0.658500      |
| 3500 | 0.642400      |



## Contributions and Support

For any inquiries, bug reports, or collaboration requests, please contact:
  
- **Email:** [Email](mailto:yusuff.baykaloglu@gmail.com)  
- **Hugging Face Profile:** [https://huggingface.co/yusufbaykaloglu](https://huggingface.co/yusufbaykaloglu)  
- **GitHub:** [yusufbaykal](https://github.com/yusufbaykal)
- **Linkedin:**[yusufbaykaloglu](https://www.linkedin.com/in/yusufbaykaloglu/)

## Citation

If you use this model in your research or project, please cite it as follows:

```bibtex
@misc{Baykaloglu2025TurkishUniversityRegulations,
  author = {Yusuf Baykaloglu},
  title = {Turkish-University-Regulations: A Fine-tuned Turkish Llama Model for University Regulations Q&A},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/yusufbaykaloglu/Turkish-University-Regulations}}
}