cointegrated commited on
Commit
3000577
1 Parent(s): f3b1f58

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +46 -5
README.md CHANGED
@@ -12,12 +12,11 @@ widget:
12
  candidate_labels: "спорт,путешествия,музыка,кино,книги,наука,политика"
13
  hypothesis_template: "Тема текста - {}."
14
  ---
15
- # RuBERT base model (cased) fine-tuned for NLI (natural language inference)
16
- The model has been trained on a series of NLI datasets automatically translated to Russian from English [from this repo](https://github.com/felipessalvatore/NLI_datasets).
17
-
18
- It predicts the logical relationship between two short texts: entailment, contradiction, or neutral.
19
 
 
20
 
 
21
  How to run the model for NLI:
22
  ```python
23
  # !pip install transformers sentencepiece --quiet
@@ -59,4 +58,46 @@ predict_zero_shot('Какая вкусная эта ваша заливная р
59
  # array([0.9059292 , 0.09407079], dtype=float32)
60
  ```
61
 
62
- Alternatively, you can use [Huggingface pipelines](https://huggingface.co/transformers/main_classes/pipelines.html) for inference.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  candidate_labels: "спорт,путешествия,музыка,кино,книги,наука,политика"
13
  hypothesis_template: "Тема текста - {}."
14
  ---
15
+ # RuBERT for NLI (natural language inference)
 
 
 
16
 
17
+ This is the [DeepPavlov/rubert-base-cased](https://huggingface.co/DeepPavlov/rubert-base-cased) fine-tuned to predict the logical relationship between two short texts: entailment, contradiction, or neutral.
18
 
19
+ ## Usage
20
  How to run the model for NLI:
21
  ```python
22
  # !pip install transformers sentencepiece --quiet
 
58
  # array([0.9059292 , 0.09407079], dtype=float32)
59
  ```
60
 
61
+ Alternatively, you can use [Huggingface pipelines](https://huggingface.co/transformers/main_classes/pipelines.html) for inference.
62
+
63
+ ## Sources
64
+ The model has been trained on a series of NLI datasets automatically translated to Russian from English.
65
+
66
+ Most datasets were taken [from the repo of Felipe Salvatore](https://github.com/felipessalvatore/NLI_datasets):
67
+ [JOCI](https://github.com/sheng-z/JOCI),
68
+ [MNLI](https://cims.nyu.edu/~sbowman/multinli/),
69
+ [MPE](https://aclanthology.org/I17-1011/),
70
+ [SICK](http://www.lrec-conf.org/proceedings/lrec2014/pdf/363_Paper.pdf),
71
+ [SNLI](https://nlp.stanford.edu/projects/snli/).
72
+
73
+ Some datasets obtained from the original sources:
74
+ [ANLI](https://github.com/facebookresearch/anli),
75
+ [NLI-style FEVER](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md),
76
+ [IMPPRES](https://github.com/facebookresearch/Imppres).
77
+
78
+ ## Performance
79
+
80
+ The table below shows ROC AUC for three models on small samples of the DEV sets:
81
+ - [tiny](https://huggingface.co/cointegrated/rubert-tiny-bilingual-nli): a small BERT predicting entailment vs not_entailment
82
+ - [twoway](https://huggingface.co/cointegrated/rubert-base-cased-nli-twoway): a base-sized BERT predicting entailment vs not_entailment
83
+ - [threeway](https://huggingface.co/cointegrated/rubert-base-cased-nli-threeway) (**this model**): a base-sized BERT predicting entailment vs contradiction vs neutral
84
+
85
+ |model |tiny/entailment|twoway/entailment|threeway/entailment|threeway[3]/contradiction|threeway[3]/neutral|
86
+ |-----------|---------------|-----------------|-------------------|-------------------------|-------------------|
87
+ |add_one_rte|0.82 |0.90 |0.92 | | |
88
+ |anli_r1 |0.50 |0.68 |0.66 |0.70 |0.75 |
89
+ |anli_r2 |0.55 |0.62 |0.62 |0.62 |0.69 |
90
+ |anli_r3 |0.50 |0.63 |0.59 |0.62 |0.64 |
91
+ |copa |0.55 |0.60 |0.62 | | |
92
+ |fever |0.88 |0.94 |0.94 |0.91 |0.92 |
93
+ |help |0.74 |0.87 |0.46 | | |
94
+ |iie |0.79 |0.85 |0.54 | | |
95
+ |imppres |0.94 |0.99 |0.99 |0.99 |0.99 |
96
+ |joci |0.87 |0.93 |0.93 |0.85 |0.80 |
97
+ |mnli |0.87 |0.92 |0.93 |0.89 |0.86 |
98
+ |monli |0.94 |1.00 |0.67 | | |
99
+ |mpe |0.82 |0.90 |0.90 |0.91 |0.80 |
100
+ |scitail |0.80 |0.96 |0.85 | | |
101
+ |sick |0.97 |0.99 |0.99 |0.98 |0.96 |
102
+ |snli |0.95 |0.98 |0.98 |0.99 |0.97 |
103
+ |terra |0.73 |0.93 |0.93 | | |