File size: 7,469 Bytes
fa15921 427da9c 9c0ae12 fa15921 3b8500d 3a32c99 fa15921 3a32c99 fa15921 3a32c99 10982b1 f1f9016 10982b1 3a32c99 10982b1 3a32c99 fa15921 10982b1 fa15921 3a32c99 fa15921 3a32c99 fa15921 10982b1 fa15921 3a32c99 fa15921 3a32c99 fa15921 878f739 fa15921 878f739 fa15921 427da9c fa15921 427da9c |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 |
---
pipeline_tag: sentence-similarity
language:
- pl
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- transformers
datasets:
- ipipan/polqa
- ipipan/maupqa
license: cc-by-sa-4.0
widget:
- source_sentence: "Pytanie: W jakim mieście urodził się Zbigniew Herbert?"
sentences:
- "Zbigniew Herbert</s>Zbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg."
- "Zbigniew Herbert</s>Lato 1968 Herbert spędził w USA (na zaproszenie Poetry Center)."
- "Herbert George Wells</s>Herbert George Wells (ur. 21 września 1866 w Bromley, zm. 13 sierpnia 1946 w Londynie) – brytyjski pisarz i biolog."
example_title: "Zbigniew Herbert"
---
# Silver Retriever Base (v1)
Silver Retriever model encodes the Polish sentences or paragraphs into a 768-dimensional dense vector space and can be used for tasks like document retrieval or semantic search.
It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/ipipan/polqa) and [MAUPQA](https://huggingface.co/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024.
## Evaluation
| **Model** | **Average [Acc]** | **Average [NDCG]** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) **[Acc]** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) **[NDCG]** | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) **[Acc]** | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) **[NDCG]** | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) **[Acc]** | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) **[NDCG]** |
|--------------------:|------------:|-------------:|------------:|-------------:|------------:|-------------:|------------:|-------------:|
| BM25 | 74.87 | 51.81 | 61.35 | 24.51 | 66.89 | 48.71 | **96.38** | **82.21** |
| BM25 (lemma) | 80.46 | 55.44 | 71.49 | 31.97 | 75.33 | 55.70 | 94.57 | 78.65 |
| [MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) | 62.62 | 39.21 | 37.24 | 11.93 | 71.67 | 51.25 | 78.97 | 54.44 |
| [LaBSE](https://huggingface.co/sentence-transformers/LaBSE) | 64.89 | 39.47 | 46.23 | 15.53 | 67.11 | 46.71 | 81.34 | 56.16 |
| [mContriever-Base](https://huggingface.co/nthakur/mcontriever-base-msmarco) | 86.31 | 60.37 | 78.66 | 36.30 | 84.44 | 67.38 | 95.82 | 77.42 |
| [E5-Base](https://huggingface.co/intfloat/multilingual-e5-base) | 91.58 | 66.56 | 86.61 | **46.08** | 91.89 | 75.90 | 96.24 | 77.69 |
| [ST-DistilRoBERTa](https://huggingface.co/sdadas/st-polish-paraphrase-from-distilroberta) | 73.78 | 48.29 | 48.43 | 16.73 | 84.89 | 64.39 | 88.02 | 63.76 |
| [ST-MPNet](sdadas/st-polish-paraphrase-from-mpnet) | 76.66 | 49.99 | 56.80 | 21.55 | 86.00 | 65.44 | 87.19 | 62.99 |
| [HerBERT-QA](https://huggingface.co/ipipan/herbert-base-qa-v1) | 84.23 | 54.36 | 75.84 | 32.52 | 85.78 | 63.58 | 91.09 | 66.99 |
| [**SilverRetriever**](https://huggingface.co/ipipan/silver-retriever-base-v1) | **92.45** | **66.72** | **87.24** | 43.40 | **94.56** | **79.66** | 95.54 | 77.10 |
Legend:
- **Acc** is the Accuracy at 10
- **NDCG** is the Normalized Discounted Cumulative Gain at 10
## Usage
### Preparing inputs
The model was trained on question-passage pairs and works best when the input is the same format as that used during training:
- We added the phrase `Pytanie:` to the beginning of the question.
- The training passages consisted of `title` and `text` concatenated with the special token `</s>`. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `</s>` token.
- Although we used the dot product during training, the model usually works better with the cosine distance.
### Inference with Sentence-Transformers
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed:
```
pip install -U sentence-transformers
```
Then you can use the model like this:
```python
from sentence_transformers import SentenceTransformer
sentences = [
"Pytanie: W jakim mieście urodził się Zbigniew Herbert?",
"Zbigniew Herbert</s>Zbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.",
]
model = SentenceTransformer('ipipan/silver-retriever-base-v1')
embeddings = model.encode(sentences)
print(embeddings)
```
### Inference with HuggingFace Transformers
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
```python
from transformers import AutoTokenizer, AutoModel
import torch
def cls_pooling(model_output, attention_mask):
return model_output[0][:,0]
# Sentences we want sentence embeddings for
sentences = [
"Pytanie: W jakim mieście urodził się Zbigniew Herbert?",
"Zbigniew Herbert</s>Zbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.",
]
# Load model from HuggingFace Hub
tokenizer = AutoTokenizer.from_pretrained('ipipan/silver-retriever-base-v1')
model = AutoModel.from_pretrained('ipipan/silver-retriever-base-v1')
# Tokenize sentences
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
# Compute token embeddings
with torch.no_grad():
model_output = model(**encoded_input)
# Perform pooling. In this case, cls pooling.
sentence_embeddings = cls_pooling(model_output, encoded_input['attention_mask'])
print("Sentence embeddings:")
print(sentence_embeddings)
```
## Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)
```
## Additional Information
### Model Creators
The model was created by Piotr Rybak from the [Institute of Computer Science, Polish Academy of Sciences](http://zil.ipipan.waw.pl/).
This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19.
### Licensing Information
CC BY-SA 4.0
### Citation Information
[More Information Needed] |