metadata

language:
  - tr
arXiv: 2403.01308
library_name: transformers
pipeline_tag: text2text-generation
inference:
  parameters:
    max_new_tokens: 32
widget:
  - text: >-
      Soru yarat: cevap: Alan Mathison Turing İngiliz matematikçi, bilgisayar
      bilimcisi ve kriptolog. II. Dünya Savaşı sırasında Alman şifrelerinin
      kırılmasında çok önemli bir rol oynadığı için savaş kahramanı sayılmıştır.
      Ayrıca Manchester Üniversitesi'nde çalıştığı yıllarda, Turing makinesi
      denilen algoritma tanımı ile modern bilgisayarların kavramsal temelini
      atmıştır.
    example_title: Question generation
  - text: >-
      Soru cevapla: Turing makinesi denilen algoritma tanımı ile modern
      bilgisayarların kavramsal temelini atan bilim insanı kimdir? kaynak: Alan
      Mathison Turing İngiliz matematikçi, bilgisayar bilimcisi ve kriptolog.
      II. Dünya Savaşı sırasında Alman şifrelerinin kırılmasında çok önemli bir
      rol oynadığı için savaş kahramanı sayılmıştır. Ayrıca Manchester
      Üniversitesi'nde çalıştığı yıllarda, Turing makinesi denilen algoritma
      tanımı ile modern bilgisayarların kavramsal temelini atmıştır.
    example_title: Question answering
  - text: >-
      yanıtları çıkar: Alan Mathison Turing İngiliz matematikçi, bilgisayar
      bilimcisi ve kriptolog.  II. Dünya Savaşı sırasında Alman şifrelerinin
      kırılmasında çok önemli bir rol oynadığı için savaş kahramanı sayılmıştır.
      <hl>  Ayrıca Manchester Üniversitesi'nde çalıştığı yıllarda, Turing
      makinesi denilen algoritma tanımı ile modern bilgisayarların kavramsal
      temelini atmıştır <hl> .
    example_title: Answer Extraction
license: cc-by-nc-sa-4.0
datasets:
  - vngrs-ai/vngrs-web-corpus

VBART Model Card

Model Description

VBART is the first sequence-to-sequence LLM pre-trained on Turkish corpora from scratch on a large scale. It was pre-trained by VNGRS in February 2023.
The model is capable of conditional text generation tasks such as text summarization, paraphrasing, and title generation when fine-tuned. It outperforms its multilingual counterparts, albeit being much smaller than other implementations.

This repository contains fine-tuned TensorFlow and Safetensors weights of VBART for question-answering and generation tasks described in the paper.

Developed by: VNGRS-AI
Model type: Transformer encoder-decoder based on mBART architecture
Language(s) (NLP): Turkish
License: CC BY-NC-SA 4.0
Finetuned from: VBART-Large
Paper: arXiv

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("vngrs-ai/VBART-Large-QAQG",
                            model_input_names=['input_ids', 'attention_mask'])
# Uncomment the device_map kwarg and delete the closing bracket to use model for inference on GPU
model = AutoModelForSeq2SeqLM.from_pretrained("vngrs-ai/VBART-Large-QAQG")#, device_map="auto")

context="..."
question="..."
highlighted_context="..."

# Prompt for question generation
qg_prompt = f'Soru yarat: cevap: {context}'
# Prompt for question answering
qa_prompt = f'Soru cevapla: {question} kaynak: {context}'
# Prompt for answer extraction
ae_prompt = f'yanıtları çıkar: {highlighted_context}'


token_input = tokenizer(ae_prompt, return_tensors="pt")#.to('cuda')
outputs = model.generate(**token_input)
print(tokenizer.decode(outputs[0]))

Training Details

Fine-tuning prompt

This model is fine-tuned on three tasks:

question answering: Answer a question in a given context. Prompted with Soru cevapla: <question> kaynak: <context>
question generation: Generate a question from a given context. Will accept a highlight token (<hl>, without spaces) to specify the answer to the question generated. Prompted with Soru yarat: <context>
answer extraction: Will extract possible answers from a highlighted range (using the same highlight token). Prompted with yanıtları çıkar: <context with highlighted parts>

Training Data

The base model is pre-trained on vngrs-web-corpus. It is curated by cleaning and filtering Turkish parts of OSCAR-2201 and mC4 datasets. These datasets consist of documents of unstructured web crawl data. More information about the dataset can be found on their respective pages. Data is filtered using a set of heuristics and certain rules, explained in the appendix of our paper.

The fine-tuning dataset is TQuAD, which has two versions. We have concatenated them and dropped duplicate samples. More information about this process can be found in Appendix B of our paper.

Limitations

This model is fine-tuned for question-answering and question-generation tasks with specific prompts. It is not intended to be used in any other case and can not be fine-tuned to any other task with full performance of the base model. It is also not guaranteed that this model will work without specified prompts.

Training Procedure

Pre-trained for 30 days and for a total of 708B tokens. Finetuned for 5 epoch.

Hardware

GPUs: 8 x Nvidia A100-80 GB

Software

TensorFlow

Hyperparameters

Pretraining

Training regime: fp16 mixed precision
Training objective: Sentence permutation and span masking (using mask lengths sampled from Poisson distribution λ=3.5, masking 30% of tokens)
Optimizer : Adam optimizer (β1 = 0.9, β2 = 0.98, Ɛ = 1e-6)
Scheduler: Custom scheduler from the original Transformers paper (20,000 warm-up steps)
Dropout: 0.1 (dropped to 0.05 and then to 0 in the last 165k and 205k steps, respectively)
Initial Learning rate: 5e-6
Training tokens: 708B

Fine-tuning

Training regime: fp16 mixed precision
Optimizer : Adam optimizer (β1 = 0.9, β2 = 0.98, Ɛ = 1e-6)
Scheduler: Linear decay scheduler
Dropout: 0.1
Learning rate: 5e-5
Fine-tune epochs: 5

Metrics

Citation

@article{turker2024vbart,
  title={VBART: The Turkish LLM},
  author={Turker, Meliksah and Ari, Erdi and Han, Aydin},
  journal={arXiv preprint arXiv:2403.01308},
  year={2024}
}