File size: 2,283 Bytes
986f7db
 
7500b48
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
986f7db
7500b48
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4a40c7e
7500b48
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
---
license: mit
datasets:
- xnli
- multi_nli
language:
- ar
- bg
- de
- el
- en
- es
- tr
- th
- ur
- hi
- zh
- vi
- fr
- ru
- sw
metrics:
- accuracy
---

# XLM-T-ROBERTA-BASE-MNLI-XNLI

## Model description 
This model takes the XLM-Roberta-base model which has been continued to pre-traine on a large corpus of Twitter in multiple languages.   
It was developed following a similar strategy as introduced as part of the [Tweet Eval](https://github.com/cardiffnlp/tweeteval) framework.   
The model is further finetuned on the MNLI dataset and also on the xnli dataset. 

## Intended Usage

This model was developed to do Zero-Shot Text Classification in the realm of Hate Speech Detection. It is finetuned on the whole xnli train set containing 15 different languages like:  
**ar, bg ,de , en, el , es, fr, hi, ru, sw, th, tr, ur, vi, zh**  
Since the base model was pre-trained on 100 different languages it has shown some effectiveness in other languages. Please refer to the list of languages in the [XLM Roberta paper](https://arxiv.org/abs/1911.02116)

### Usage with Zero-Shot Classification pipeline
```python
from transformers import pipeline
classifier = pipeline("zero-shot-classification",
                      model="morit/xlm-t-roberta-base-mnli-xnli")
```


## Training 

This model was pre-trained on set of 100 languages, as described in the original paper. It was then fine-tuned on the task of NLI on the concatenated MNLI train set. Finally, it was trained for one additional epoch on only XNLI data where the translations for the premise and hypothesis are shuffled such that the premise and hypothesis for each example come from the same original English example but the premise and hypothesis are of different languages.
The following hyper-parameters were chosen:
- learning rate:    2e-5
- batch size:     32
- max sequence:   length 128

using one GPU (NVIDIA GeForce RTX 3090)

## Evaluation 
The model was evaluated on all the test sets of the xnli dataset resulting in the following accuracies: 

|  ar |  bg |  de |  el | en | es |  fr |  hi|  ru | sw | th | tr |ur | vi | zh |
|-----|-----|-----|----|----|----|----|----|----|----|----|----|----|----|----|
|0.776|0.804|0.796|0.791|0.851|0.813|0.806|0.757|0.783|0.716|0.765|0.780|0.705|0.795|0.782|