Create README.md

README.md

@@ -0,0 +1,67 @@
+## Persian XLM-RoBERTa Large For Qusetion Answering Task
+
+XLM-RoBERTa is a multilingual language model pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It was introduced in the paper [Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116v2) by Conneau et al. .
+
+Multilingual [XLM-RoBERTa large for QA on various languages](https://huggingface.co/deepset/xlm-roberta-large-squad2) is fine-tuned on various QA datasets but PQuAD, which is the biggest persian QA dataset so far. This second model is our base model to be fine-tuned.
+
+Paper presenting PQuAD dataset: [arXiv:2202.06219](https://arxiv.org/abs/2202.06219)
+
+---
+
+## Introduction
+
+This model is fine-tuned on PQuAD Train set and is easily ready to use. Too long training time encouraged me to publish this model in order to make life easier for those who need.
+
+
+## Hyperparameters
+The reason I set batch_size to 4 is limited GPU memory in Google Colab.
+Pre-processing, training and evaluating model took me about 4 hours.
+```
+batch_size = 4
+n_epochs = 1
+base_LM_model = "deepset/xlm-roberta-large-squad2"
+max_seq_len = 256
+learning_rate = 3e-5
+evaluation_strategy = "epoch",
+save_strategy = "epoch",
+learning_rate = 3e-5,
+warmup_ratio = 0.1,
+gradient_accumulation_steps = 8,
+weight_decay = 0.01,
+``` 
+## Performance
+Evaluated on the PQuAD Persian test set with the [official PQuAD link](https://huggingface.co/datasets/newsha/PQuAD/blob/main/Test_v8.0.json).
+I trained for more than 1 epoch as well, but I get worse results.
+```
+  "exact_match": 66.85,
+  "f1": 87.56
+```
+## How to use
+```python
+from transformers import AutoTokenizer, AutoModelForQuestionAnswering
+tokenizer = AutoTokenizer.from_pretrained('pedramyazdipoor/persian_xlm_roberta_large')
+model = AutoModel.from_pretrained('pedramyazdipoor/persian_xlm_roberta_large')
+device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+model.eval().to(device)
+text = 'سلام من پدرامم 26 سالمه'
+question = 'نامم چیست؟'
+print(tokenizer.tokenize(text + question))
+encoding = tokenizer(text,question,add_special_tokens = True,
+                     return_token_type_ids = True,
+                     return_tensors = 'pt',
+                     padding = True,
+                     return_offsets_mapping = True,
+                     truncation = 'only_first',
+                     max_length = 32)
+out = model(encoding['input_ids'].to(device),encoding['attention_mask'].to(device), encoding['token_type_ids'].to(device))
+#out.start_logits
+#out.end_logits
+>>> ['▁سلام', '▁من', '▁پدر', 'ام', 'م', '▁26', '▁سالم', 'ه', 'نام', 'م', '▁چیست', '؟']
+```
+## Acknowledgments
+We hereby, express our gratitude to the [Newsha Shahbodaghkhan](https://huggingface.co/datasets/newsha/PQuAD/tree/main) for facilitating dataset gathering.
+## Contributors
+- Pedram Yazdipoor : [Linkedin](https://www.linkedin.com/in/pedram-yazdipour/)
+## Releases
+### Release v0.1 (Sep 14, 2022)
+This is the first version of our Persian XLM-Roberta-Large.