|
--- |
|
pipeline_tag: sentence-similarity |
|
language: |
|
- pl |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
- transformers |
|
datasets: |
|
- ipipan/polqa |
|
- ipipan/maupqa |
|
license: cc-by-sa-4.0 |
|
widget: |
|
- source_sentence: "Pytanie: W jakim mieście urodził się Zbigniew Herbert?" |
|
sentences: |
|
- "Zbigniew Herbert</s>Zbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg." |
|
- "Zbigniew Herbert</s>Lato 1968 Herbert spędził w USA (na zaproszenie Poetry Center)." |
|
- "Herbert George Wells</s>Herbert George Wells (ur. 21 września 1866 w Bromley, zm. 13 sierpnia 1946 w Londynie) – brytyjski pisarz i biolog." |
|
example_title: "Zbigniew Herbert" |
|
--- |
|
|
|
 |
|
|
|
# Silver Retriever Base (v1) |
|
|
|
Silver Retriever model encodes the Polish sentences or paragraphs into a 768-dimensional dense vector space and can be used for tasks like document retrieval or semantic search. |
|
|
|
It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/ipipan/polqa) and [MAUPQA](https://huggingface.co/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024. Please refer to the [SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering](https://arxiv.org/abs/2309.08469) for more details. |
|
|
|
## Evaluation |
|
|
|
|
|
| **Model** | **Average [Acc]** | **Average [NDCG]** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) **[Acc]** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) **[NDCG]** | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) **[Acc]** | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) **[NDCG]** | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) **[Acc]** | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) **[NDCG]** | |
|
|--------------------:|------------:|-------------:|------------:|-------------:|------------:|-------------:|------------:|-------------:| |
|
| BM25 | 74.87 | 51.81 | 61.35 | 24.51 | 66.89 | 48.71 | 96.38 | **82.21** | |
|
| BM25 (lemma) | 80.46 | 55.44 | 71.49 | 31.97 | 75.33 | 55.70 | 94.57 | 78.65 | |
|
| [MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) | 62.62 | 39.21 | 37.24 | 11.93 | 71.67 | 51.25 | 78.97 | 54.44 | |
|
| [LaBSE](https://huggingface.co/sentence-transformers/LaBSE) | 64.89 | 39.47 | 46.23 | 15.53 | 67.11 | 46.71 | 81.34 | 56.16 | |
|
| [mContriever-Base](https://huggingface.co/nthakur/mcontriever-base-msmarco) | 86.31 | 60.37 | 78.66 | 36.30 | 84.44 | 67.38 | 95.82 | 77.42 | |
|
| [E5-Base](https://huggingface.co/intfloat/multilingual-e5-base) | 91.58 | 66.56 | 86.61 | **46.08** | 91.89 | 75.90 | 96.24 | 77.69 | |
|
| [ST-DistilRoBERTa](https://huggingface.co/sdadas/st-polish-paraphrase-from-distilroberta) | 73.78 | 48.29 | 48.43 | 16.73 | 84.89 | 64.39 | 88.02 | 63.76 | |
|
| [ST-MPNet](https://huggingface.co/sdadas/st-polish-paraphrase-from-mpnet) | 76.66 | 49.99 | 56.80 | 21.55 | 86.00 | 65.44 | 87.19 | 62.99 | |
|
| [HerBERT-QA](https://huggingface.co/ipipan/herbert-base-qa-v1) | 84.23 | 54.36 | 75.84 | 32.52 | 85.78 | 63.58 | 91.09 | 66.99 | |
|
| [Silver Retriever v1](https://huggingface.co/ipipan/silver-retriever-base-v1) | 92.45 | 66.72 | 87.24 | 43.40 | **94.56** | 79.66 | 95.54 | 77.10 | |
|
| [Silver Retriever v1.1](https://huggingface.co/ipipan/silver-retriever-base-v1.1) | **93.18** | **67.55** | **88.60** | 44.88 | 94.00 | **79.83** | **96.94** | 77.95 | |
|
|
|
Legend: |
|
- **Acc** is the Accuracy at 10 |
|
- **NDCG** is the Normalized Discounted Cumulative Gain at 10 |
|
|
|
|
|
## Usage |
|
|
|
### Preparing inputs |
|
|
|
The model was trained on question-passage pairs and works best when the input is the same format as that used during training: |
|
- We added the phrase `Pytanie:` to the beginning of the question. |
|
- The training passages consisted of `title` and `text` concatenated with the special token `</s>`. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `</s>` token. |
|
- Although we used the dot product during training, the model usually works better with the cosine distance. |
|
|
|
### Inference with Sentence-Transformers |
|
|
|
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: |
|
|
|
``` |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
Then you can use the model like this: |
|
|
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
sentences = [ |
|
"Pytanie: W jakim mieście urodził się Zbigniew Herbert?", |
|
"Zbigniew Herbert</s>Zbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.", |
|
] |
|
|
|
model = SentenceTransformer('ipipan/silver-retriever-base-v1') |
|
embeddings = model.encode(sentences) |
|
print(embeddings) |
|
``` |
|
|
|
### Inference with HuggingFace Transformers |
|
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings. |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
import torch |
|
|
|
|
|
def cls_pooling(model_output, attention_mask): |
|
return model_output[0][:,0] |
|
|
|
|
|
# Sentences we want sentence embeddings for |
|
sentences = [ |
|
"Pytanie: W jakim mieście urodził się Zbigniew Herbert?", |
|
"Zbigniew Herbert</s>Zbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.", |
|
] |
|
# Load model from HuggingFace Hub |
|
tokenizer = AutoTokenizer.from_pretrained('ipipan/silver-retriever-base-v1') |
|
model = AutoModel.from_pretrained('ipipan/silver-retriever-base-v1') |
|
|
|
# Tokenize sentences |
|
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') |
|
|
|
# Compute token embeddings |
|
with torch.no_grad(): |
|
model_output = model(**encoded_input) |
|
|
|
# Perform pooling. In this case, cls pooling. |
|
sentence_embeddings = cls_pooling(model_output, encoded_input['attention_mask']) |
|
|
|
print("Sentence embeddings:") |
|
print(sentence_embeddings) |
|
``` |
|
|
|
## Full Model Architecture |
|
``` |
|
SentenceTransformer( |
|
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel |
|
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) |
|
) |
|
``` |
|
|
|
## Additional Information |
|
|
|
### Model Creators |
|
|
|
The model was created by Piotr Rybak from the [Institute of Computer Science, Polish Academy of Sciences](http://zil.ipipan.waw.pl/). |
|
|
|
This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19. |
|
|
|
### Licensing Information |
|
|
|
CC BY-SA 4.0 |
|
|
|
### Citation Information |
|
|
|
``` |
|
@misc{rybak2023silverretriever, |
|
title={SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering}, |
|
author={Piotr Rybak and Maciej Ogrodniczuk}, |
|
year={2023}, |
|
eprint={2309.08469}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
