--- pipeline_tag: sentence-similarity language: - pl tags: - sentence-transformers - feature-extraction - sentence-similarity - transformers datasets: - ipipan/polqa - ipipan/maupqa license: cc-by-sa-4.0 widget: - source_sentence: "Pytanie: W jakim mieście urodził się Zbigniew Herbert?" sentences: - "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg." - "Zbigniew HerbertLato 1968 Herbert spędził w USA (na zaproszenie Poetry Center)." - "Herbert George WellsHerbert George Wells (ur. 21 września 1866 w Bromley, zm. 13 sierpnia 1946 w Londynie) – brytyjski pisarz i biolog." example_title: "Zbigniew Herbert" --- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/5eb2c5ef4e876668a0c3779e/j2JE7_VnbRifCmV7_4BP9.png) # Silver Retriever Base (v1) Silver Retriever model encodes the Polish sentences or paragraphs into a 768-dimensional dense vector space and can be used for tasks like document retrieval or semantic search. It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/ipipan/polqa) and [MAUPQA](https://huggingface.co/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024. Please refer to the [SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering](https://arxiv.org/abs/2309.08469) for more details. ## Evaluation | **Model** | **Average [Acc]** | **Average [NDCG]** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) **[Acc]** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) **[NDCG]** | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) **[Acc]** | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) **[NDCG]** | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) **[Acc]** | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) **[NDCG]** | |--------------------:|------------:|-------------:|------------:|-------------:|------------:|-------------:|------------:|-------------:| | BM25 | 74.87 | 51.81 | 61.35 | 24.51 | 66.89 | 48.71 | **96.38** | **82.21** | | BM25 (lemma) | 80.46 | 55.44 | 71.49 | 31.97 | 75.33 | 55.70 | 94.57 | 78.65 | | [MiniLM-L12-v2](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2) | 62.62 | 39.21 | 37.24 | 11.93 | 71.67 | 51.25 | 78.97 | 54.44 | | [LaBSE](https://huggingface.co/sentence-transformers/LaBSE) | 64.89 | 39.47 | 46.23 | 15.53 | 67.11 | 46.71 | 81.34 | 56.16 | | [mContriever-Base](https://huggingface.co/nthakur/mcontriever-base-msmarco) | 86.31 | 60.37 | 78.66 | 36.30 | 84.44 | 67.38 | 95.82 | 77.42 | | [E5-Base](https://huggingface.co/intfloat/multilingual-e5-base) | 91.58 | 66.56 | 86.61 | **46.08** | 91.89 | 75.90 | 96.24 | 77.69 | | [ST-DistilRoBERTa](https://huggingface.co/sdadas/st-polish-paraphrase-from-distilroberta) | 73.78 | 48.29 | 48.43 | 16.73 | 84.89 | 64.39 | 88.02 | 63.76 | | [ST-MPNet](https://huggingface.co/sdadas/st-polish-paraphrase-from-mpnet) | 76.66 | 49.99 | 56.80 | 21.55 | 86.00 | 65.44 | 87.19 | 62.99 | | [HerBERT-QA](https://huggingface.co/ipipan/herbert-base-qa-v1) | 84.23 | 54.36 | 75.84 | 32.52 | 85.78 | 63.58 | 91.09 | 66.99 | | [**Silver Retriever v1**](https://huggingface.co/ipipan/silver-retriever-base-v1) | **92.45** | **66.72** | **87.24** | 43.40 | **94.56** | **79.66** | 95.54 | 77.10 | Legend: - **Acc** is the Accuracy at 10 - **NDCG** is the Normalized Discounted Cumulative Gain at 10 ## Usage ### Preparing inputs The model was trained on question-passage pairs and works best when the input is the same format as that used during training: - We added the phrase `Pytanie:` to the beginning of the question. - The training passages consisted of `title` and `text` concatenated with the special token ``. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `` token. - Although we used the dot product during training, the model usually works better with the cosine distance. ### Inference with Sentence-Transformers Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: ``` pip install -U sentence-transformers ``` Then you can use the model like this: ```python from sentence_transformers import SentenceTransformer sentences = [ "Pytanie: W jakim mieście urodził się Zbigniew Herbert?", "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.", ] model = SentenceTransformer('ipipan/silver-retriever-base-v1') embeddings = model.encode(sentences) print(embeddings) ``` ### Inference with HuggingFace Transformers Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings. ```python from transformers import AutoTokenizer, AutoModel import torch def cls_pooling(model_output, attention_mask): return model_output[0][:,0] # Sentences we want sentence embeddings for sentences = [ "Pytanie: W jakim mieście urodził się Zbigniew Herbert?", "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.", ] # Load model from HuggingFace Hub tokenizer = AutoTokenizer.from_pretrained('ipipan/silver-retriever-base-v1') model = AutoModel.from_pretrained('ipipan/silver-retriever-base-v1') # Tokenize sentences encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') # Compute token embeddings with torch.no_grad(): model_output = model(**encoded_input) # Perform pooling. In this case, cls pooling. sentence_embeddings = cls_pooling(model_output, encoded_input['attention_mask']) print("Sentence embeddings:") print(sentence_embeddings) ``` ## Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) ) ``` ## Additional Information ### Model Creators The model was created by Piotr Rybak from the [Institute of Computer Science, Polish Academy of Sciences](http://zil.ipipan.waw.pl/). This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19. ### Licensing Information CC BY-SA 4.0 ### Citation Information ``` @misc{rybak2023silverretriever, title={SilverRetriever: Advancing Neural Passage Retrieval for Polish Question Answering}, author={Piotr Rybak and Maciej Ogrodniczuk}, year={2023}, eprint={2309.08469}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```