Use FP32 metrics
Browse files
README.md
CHANGED
@@ -26,7 +26,7 @@ The model was trained and tested in the following languages:
|
|
26 |
|
27 |
| Metric | Value |
|
28 |
|:--------------------|------:|
|
29 |
-
| Relevance (NDCG@10) | 0.
|
30 |
|
31 |
Note that the relevance score is computed as an average over 14 retrieval datasets (see
|
32 |
[details below](#evaluation-metrics)).
|
@@ -35,12 +35,11 @@ Note that the relevance score is computed as an average over 14 retrieval datase
|
|
35 |
|
36 |
| GPU | Batch size 32 |
|
37 |
|:-----------|--------------:|
|
38 |
-
| NVIDIA A10 |
|
39 |
-
| NVIDIA T4 |
|
40 |
|
41 |
The inference times only measure the time the model takes to process a single batch, it does not include pre- or
|
42 |
-
post-processing steps like the tokenization.
|
43 |
-
[FP16](https://en.wikipedia.org/wiki/Half-precision_floating-point_format) version of the model.
|
44 |
|
45 |
## Requirements
|
46 |
|
@@ -77,22 +76,22 @@ To determine the relevance score, we averaged the results that we obtained when
|
|
77 |
|
78 |
| Dataset | NDCG@10 |
|
79 |
|:------------------|--------:|
|
80 |
-
| Average | 0.
|
81 |
| | |
|
82 |
-
| Arguana | 0.
|
83 |
| CLIMATE-FEVER | 0.159 |
|
84 |
| DBPedia Entity | 0.355 |
|
85 |
-
| FEVER | 0.
|
86 |
| FiQA-2018 | 0.282 |
|
87 |
| HotpotQA | 0.688 |
|
88 |
-
| MS MARCO | 0.
|
89 |
| NFCorpus | 0.341 |
|
90 |
-
| NQ | 0.
|
91 |
-
| Quora | 0.
|
92 |
| SCIDOCS | 0.143 |
|
93 |
-
| SciFact | 0.
|
94 |
-
| TREC-COVID | 0.
|
95 |
-
| Webis-Touche-2020 | 0.
|
96 |
|
97 |
We evaluated the model on the datasets of the [MIRACL benchmark](https://github.com/project-miracl/miracl) to test its
|
98 |
multilingual capacities. Note that not all training languages are part of the benchmark, so we only report the metrics
|
@@ -100,6 +99,6 @@ for the existing languages.
|
|
100 |
|
101 |
| Language | NDCG@10 |
|
102 |
|:---------|--------:|
|
103 |
-
| French | 0.
|
104 |
-
| German | 0.
|
105 |
-
| Spanish | 0.
|
|
|
26 |
|
27 |
| Metric | Value |
|
28 |
|:--------------------|------:|
|
29 |
+
| Relevance (NDCG@10) | 0.453 |
|
30 |
|
31 |
Note that the relevance score is computed as an average over 14 retrieval datasets (see
|
32 |
[details below](#evaluation-metrics)).
|
|
|
35 |
|
36 |
| GPU | Batch size 32 |
|
37 |
|:-----------|--------------:|
|
38 |
+
| NVIDIA A10 | 8 ms |
|
39 |
+
| NVIDIA T4 | 21 ms |
|
40 |
|
41 |
The inference times only measure the time the model takes to process a single batch, it does not include pre- or
|
42 |
+
post-processing steps like the tokenization.
|
|
|
43 |
|
44 |
## Requirements
|
45 |
|
|
|
76 |
|
77 |
| Dataset | NDCG@10 |
|
78 |
|:------------------|--------:|
|
79 |
+
| Average | 0.453 |
|
80 |
| | |
|
81 |
+
| Arguana | 0.516 |
|
82 |
| CLIMATE-FEVER | 0.159 |
|
83 |
| DBPedia Entity | 0.355 |
|
84 |
+
| FEVER | 0.729 |
|
85 |
| FiQA-2018 | 0.282 |
|
86 |
| HotpotQA | 0.688 |
|
87 |
+
| MS MARCO | 0.334 |
|
88 |
| NFCorpus | 0.341 |
|
89 |
+
| NQ | 0.438 |
|
90 |
+
| Quora | 0.726 |
|
91 |
| SCIDOCS | 0.143 |
|
92 |
+
| SciFact | 0.630 |
|
93 |
+
| TREC-COVID | 0.664 |
|
94 |
+
| Webis-Touche-2020 | 0.337 |
|
95 |
|
96 |
We evaluated the model on the datasets of the [MIRACL benchmark](https://github.com/project-miracl/miracl) to test its
|
97 |
multilingual capacities. Note that not all training languages are part of the benchmark, so we only report the metrics
|
|
|
99 |
|
100 |
| Language | NDCG@10 |
|
101 |
|:---------|--------:|
|
102 |
+
| French | 0.346 |
|
103 |
+
| German | 0.368 |
|
104 |
+
| Spanish | 0.416 |
|