saattrupdan
commited on
Commit
•
24b78a4
1
Parent(s):
e06ebc7
Update README.md
Browse files
README.md
CHANGED
@@ -77,12 +77,13 @@ The Scandinavian scores are the average of the Danish, Swedish and Norwegian sco
|
|
77 |
|
78 |
| **Model** | **MCC** | **Macro-F1** | **Accuracy** | **Number of Parameters** |
|
79 |
| :-------- | :------------ | :--------- | :----------- | :----------- |
|
80 |
-
| `alexandrainst/scandi-nli-large` (this) |
|
81 |
| [`MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7) | 69.01% | 71.99% | 80.66% | 279M |
|
82 |
-
| [`alexandrainst/scandi-nli-base`](https://huggingface.co/alexandrainst/scandi-nli-base) |
|
|
|
83 |
| [`MoritzLaurer/mDeBERTa-v3-base-mnli-xnli`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) | 63.94% | 70.41% | 77.23% | 279M |
|
84 |
-
| [`
|
85 |
-
| [`alexandrainst/scandi-nli-small`](https://huggingface.co/alexandrainst/scandi-nli-small) |
|
86 |
|
87 |
|
88 |
### Danish Evaluation
|
@@ -96,6 +97,7 @@ We use a test split of the [DanFEVER dataset](https://aclanthology.org/2021.noda
|
|
96 |
| [`alexandrainst/scandi-nli-base`](https://huggingface.co/alexandrainst/scandi-nli-base) | 62.44% | 55.00% | 80.42% | 178M |
|
97 |
| [`MoritzLaurer/mDeBERTa-v3-base-mnli-xnli`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) | 52.79% | 52.00% | 72.35% | 279M |
|
98 |
| [`joeddav/xlm-roberta-large-xnli`](https://huggingface.co/joeddav/xlm-roberta-large-xnli) | 49.18% | 50.31% | 69.73% | 560M |
|
|
|
99 |
| [`alexandrainst/scandi-nli-small`](https://huggingface.co/alexandrainst/scandi-nli-small) | 47.28% | 48.88% | 73.46% | **22M** |
|
100 |
|
101 |
|
@@ -107,12 +109,13 @@ We acknowledge that not evaluating on a gold standard dataset is not ideal, but
|
|
107 |
|
108 |
| **Model** | **MCC** | **Macro-F1** | **Accuracy** | **Number of Parameters** |
|
109 |
| :-------- | :------------ | :--------- | :----------- | :----------- |
|
110 |
-
| `alexandrainst/scandi-nli-large` (this) |
|
111 |
-
| [`
|
112 |
| [`MoritzLaurer/mDeBERTa-v3-base-mnli-xnli`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) | 73.84% | 82.46% | 82.58% | 279M |
|
113 |
| [`MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7) | 73.32% | 82.15% | 82.08% | 279M |
|
114 |
-
| [`
|
115 |
-
| [`
|
|
|
116 |
|
117 |
|
118 |
### Norwegian Evaluation
|
@@ -123,13 +126,13 @@ We acknowledge that not evaluating on a gold standard dataset is not ideal, but
|
|
123 |
|
124 |
| **Model** | **MCC** | **Macro-F1** | **Accuracy** | **Number of Parameters** |
|
125 |
| :-------- | :------------ | :--------- | :----------- | :----------- |
|
126 |
-
| `alexandrainst/scandi-nli-large` (this) |
|
|
|
|
|
127 |
| [`MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7) | 65.33% | 76.73% | 76.65% | 279M |
|
128 |
-
| [`alexandrainst/scandi-nli-base`](https://huggingface.co/alexandrainst/scandi-nli-base) | asd | asd | asd | 178M |
|
129 |
-
| [`NbAiLab/nb-bert-base-mnli`](https://huggingface.co/NbAiLab/nb-bert-base-mnli) | asd | asd | asd | 178M |
|
130 |
| [`MoritzLaurer/mDeBERTa-v3-base-mnli-xnli`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) | 65.18% | 76.76% | 76.77% | 279M |
|
131 |
-
| [`
|
132 |
-
| [`alexandrainst/scandi-nli-small`](https://huggingface.co/alexandrainst/scandi-nli-small) |
|
133 |
|
134 |
|
135 |
## Training procedure
|
|
|
77 |
|
78 |
| **Model** | **MCC** | **Macro-F1** | **Accuracy** | **Number of Parameters** |
|
79 |
| :-------- | :------------ | :--------- | :----------- | :----------- |
|
80 |
+
| `alexandrainst/scandi-nli-large` (this) | **73.70%** | **74.44%** | **83.91%** | 354M |
|
81 |
| [`MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7) | 69.01% | 71.99% | 80.66% | 279M |
|
82 |
+
| [`alexandrainst/scandi-nli-base`](https://huggingface.co/alexandrainst/scandi-nli-base) | 67.42% | 71.54% | 80.09% | 178M |
|
83 |
+
| [`joeddav/xlm-roberta-large-xnli`](https://huggingface.co/joeddav/xlm-roberta-large-xnli) | 64.17% | 70.80% | 77.29% | 560M |
|
84 |
| [`MoritzLaurer/mDeBERTa-v3-base-mnli-xnli`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) | 63.94% | 70.41% | 77.23% | 279M |
|
85 |
+
| [`NbAiLab/nb-bert-base-mnli`](https://huggingface.co/NbAiLab/nb-bert-base-mnli) | 61.71% | 68.36% | 76.08% | 178M |
|
86 |
+
| [`alexandrainst/scandi-nli-small`](https://huggingface.co/alexandrainst/scandi-nli-small) | 56.02% | 65.30% | 73.56% | **22M** |
|
87 |
|
88 |
|
89 |
### Danish Evaluation
|
|
|
97 |
| [`alexandrainst/scandi-nli-base`](https://huggingface.co/alexandrainst/scandi-nli-base) | 62.44% | 55.00% | 80.42% | 178M |
|
98 |
| [`MoritzLaurer/mDeBERTa-v3-base-mnli-xnli`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) | 52.79% | 52.00% | 72.35% | 279M |
|
99 |
| [`joeddav/xlm-roberta-large-xnli`](https://huggingface.co/joeddav/xlm-roberta-large-xnli) | 49.18% | 50.31% | 69.73% | 560M |
|
100 |
+
| [`NbAiLab/nb-bert-base-mnli`](https://huggingface.co/NbAiLab/nb-bert-base-mnli) | 56.92% | 53.25% | 76.39% | 178M |
|
101 |
| [`alexandrainst/scandi-nli-small`](https://huggingface.co/alexandrainst/scandi-nli-small) | 47.28% | 48.88% | 73.46% | **22M** |
|
102 |
|
103 |
|
|
|
109 |
|
110 |
| **Model** | **MCC** | **Macro-F1** | **Accuracy** | **Number of Parameters** |
|
111 |
| :-------- | :------------ | :--------- | :----------- | :----------- |
|
112 |
+
| `alexandrainst/scandi-nli-large` (this) | **76.69%** | **84.47%** | **84.38%** | 354M |
|
113 |
+
| [`joeddav/xlm-roberta-large-xnli`](https://huggingface.co/joeddav/xlm-roberta-large-xnli) | 75.35% | 83.42% | 83.55% | 560M |
|
114 |
| [`MoritzLaurer/mDeBERTa-v3-base-mnli-xnli`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) | 73.84% | 82.46% | 82.58% | 279M |
|
115 |
| [`MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7) | 73.32% | 82.15% | 82.08% | 279M |
|
116 |
+
| [`alexandrainst/scandi-nli-base`](https://huggingface.co/alexandrainst/scandi-nli-base) | 72.29% | 81.37% | 81.51% | 178M |
|
117 |
+
| [`NbAiLab/nb-bert-base-mnli`](https://huggingface.co/NbAiLab/nb-bert-base-mnli) | 64.69% | 76.40% | 76.47% | 178M |
|
118 |
+
| [`alexandrainst/scandi-nli-small`](https://huggingface.co/alexandrainst/scandi-nli-small) | 62.35% | 74.79% | 74.93% | **22M** |
|
119 |
|
120 |
|
121 |
### Norwegian Evaluation
|
|
|
126 |
|
127 |
| **Model** | **MCC** | **Macro-F1** | **Accuracy** | **Number of Parameters** |
|
128 |
| :-------- | :------------ | :--------- | :----------- | :----------- |
|
129 |
+
| `alexandrainst/scandi-nli-large` (this) | **70.61%** | **80.43%** | **80.36%** | 354M |
|
130 |
+
| [`joeddav/xlm-roberta-large-xnli`](https://huggingface.co/joeddav/xlm-roberta-large-xnli) | 67.99% | 78.68% | 78.60% | 560M |
|
131 |
+
| [`alexandrainst/scandi-nli-base`](https://huggingface.co/alexandrainst/scandi-nli-base) | 67.53% | 78.24% | 78.33% | 178M |
|
132 |
| [`MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-xnli-multilingual-nli-2mil7) | 65.33% | 76.73% | 76.65% | 279M |
|
|
|
|
|
133 |
| [`MoritzLaurer/mDeBERTa-v3-base-mnli-xnli`](https://huggingface.co/MoritzLaurer/mDeBERTa-v3-base-mnli-xnli) | 65.18% | 76.76% | 76.77% | 279M |
|
134 |
+
| [`NbAiLab/nb-bert-base-mnli`](https://huggingface.co/NbAiLab/nb-bert-base-mnli) | 63.51% | 75.42% | 75.39% | 178M |
|
135 |
+
| [`alexandrainst/scandi-nli-small`](https://huggingface.co/alexandrainst/scandi-nli-small) | 58.42% | 72.22% | 72.30% | **22M** |
|
136 |
|
137 |
|
138 |
## Training procedure
|