Commit
•
10982b1
1
Parent(s):
9c0ae12
add evaluation
Browse files
README.md
CHANGED
|
@@ -26,14 +26,51 @@ Silver Retriever model encodes the Polish sentences or paragraphs into a 768-dim
|
|
| 26 |
|
| 27 |
It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/ipipan/polqa) and [MAUPQA](https://huggingface.co/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024.
|
| 28 |
|
| 29 |
-
##
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
|
| 31 |
The model was trained on question-passage pairs and works best when the input is the same format as that used during training:
|
| 32 |
-
- We added the phrase `Pytanie
|
| 33 |
- The training passages consisted of `title` and `text` concatenated with the special token `</s>`. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `</s>` token.
|
| 34 |
- Although we used the dot product during training, the model usually works better with the cosine distance.
|
| 35 |
|
| 36 |
-
|
| 37 |
|
| 38 |
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
|
| 39 |
|
|
@@ -55,7 +92,7 @@ embeddings = model.encode(sentences)
|
|
| 55 |
print(embeddings)
|
| 56 |
```
|
| 57 |
|
| 58 |
-
|
| 59 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
| 60 |
|
| 61 |
```python
|
|
|
|
| 26 |
|
| 27 |
It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/ipipan/polqa) and [MAUPQA](https://huggingface.co/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024.
|
| 28 |
|
| 29 |
+
## Evaluation
|
| 30 |
+
|
| 31 |
+
|
| 32 |
+
### Accuracy@10
|
| 33 |
+
|
| 34 |
+
| **Model** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) | **Average** |
|
| 35 |
+
|:-----------------------|------------:|------------:|------------:|------------:|
|
| 36 |
+
| BM25 | 61.35 | 66.89 | **96.38** | 74.87 |
|
| 37 |
+
| BM25 (lemma) | 71.49 | 75.33 | 94.57 | 80.46 |
|
| 38 |
+
| MiniLM-L12-v2 | 37.24 | 71.67 | 78.97 | 62.62 |
|
| 39 |
+
| LaBSE | 46.23 | 67.11 | 81.34 | 64.89 |
|
| 40 |
+
| mContriever-Base | 78.66 | 84.44 | 95.82 | 86.31 |
|
| 41 |
+
| E5-Base | 86.61 | 91.89 | 96.24 | 91.58 |
|
| 42 |
+
| ST-DistilRoBERTa | 48.43 | 84.89 | 88.02 | 73.78 |
|
| 43 |
+
| ST-MPNet | 56.80 | 86.00 | 87.19 | 76.66 |
|
| 44 |
+
| HerBERT-QA | 75.84 | 85.78 | 91.09 | 84.23 |
|
| 45 |
+
| **SilverRetriever** | **87.24** | **94.56** | 95.54 | **92.45** |
|
| 46 |
+
|
| 47 |
+
### NDCG@10
|
| 48 |
+
|
| 49 |
+
| **Model** | **PolQA** | **Allegro FAQ** | **Legal Questions** | **Average** |
|
| 50 |
+
|:-----------------------|-------------:|-------------:|-------------:|-------------:|
|
| 51 |
+
| BM25 | 24.51 | 48.71 | **82.21** | 51.81 |
|
| 52 |
+
| BM25 (lemma) | 31.97 | 55.70 | 78.65 | 55.44 |
|
| 53 |
+
| MiniLM-L12-v2 | 11.93 | 51.25 | 54.44 | 39.21 |
|
| 54 |
+
| LaBSE | 15.53 | 46.71 | 56.16 | 39.47 |
|
| 55 |
+
| mContriever-Base | 36.30 | 67.38 | 77.42 | 60.37 |
|
| 56 |
+
| E5-Base | **46.08** | 75.90 | 77.69 | 66.56 |
|
| 57 |
+
| ST-DistilRoBERTa | 16.73 | 64.39 | 63.76 | 48.29 |
|
| 58 |
+
| ST-MPNet | 21.55 | 65.44 | 62.99 | 49.99 |
|
| 59 |
+
| HerBERT-QA | 32.52 | 63.58 | 66.99 | 54.36 |
|
| 60 |
+
| **SilverRetriever** | 43.40 | **79.66** | 77.10 | **66.72** |
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
|
| 64 |
+
## Usage
|
| 65 |
+
|
| 66 |
+
### Preparing inputs
|
| 67 |
|
| 68 |
The model was trained on question-passage pairs and works best when the input is the same format as that used during training:
|
| 69 |
+
- We added the phrase `Pytanie:` to the beginning of the question.
|
| 70 |
- The training passages consisted of `title` and `text` concatenated with the special token `</s>`. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `</s>` token.
|
| 71 |
- Although we used the dot product during training, the model usually works better with the cosine distance.
|
| 72 |
|
| 73 |
+
### Inference with Sentence-Transformers
|
| 74 |
|
| 75 |
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
|
| 76 |
|
|
|
|
| 92 |
print(embeddings)
|
| 93 |
```
|
| 94 |
|
| 95 |
+
### Inference with HuggingFace Transformers
|
| 96 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
| 97 |
|
| 98 |
```python
|