--- pipeline_tag: sentence-similarity language: - pl tags: - sentence-transformers - feature-extraction - sentence-similarity - transformers datasets: - ipipan/polqa - ipipan/maupqa license: cc-by-sa-4.0 widget: - source_sentence: "Pytanie: W jakim mieście urodził się Zbigniew Herbert?" sentences: - "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg." - "Zbigniew HerbertLato 1968 Herbert spędził w USA (na zaproszenie Poetry Center)." - "Herbert George WellsHerbert George Wells (ur. 21 września 1866 w Bromley, zm. 13 sierpnia 1946 w Londynie) – brytyjski pisarz i biolog." example_title: "Zbigniew Herbert" --- # Silver Retriever Base (v1) Silver Retriever model encodes the Polish sentences or paragraphs into a 768-dimensional dense vector space and can be used for tasks like document retrieval or semantic search. It was initialized from the [HerBERT-base](https://huggingface.co/allegro/herbert-base-cased) model and fine-tuned on the [PolQA](https://huggingface.co/ipipan/polqa) and [MAUPQA](https://huggingface.co/ipipan/maupqa) datasets for 15,000 steps with a batch size of 1,024. ## Evaluation ### Accuracy@10 | **Model** | [**PolQA**](https://huggingface.co/datasets/ipipan/polqa) | [**Allegro FAQ**](https://huggingface.co/datasets/piotr-rybak/allegro-faq) | [**Legal Questions**](https://huggingface.co/datasets/piotr-rybak/legal-questions) | **Average** | |:-----------------------|------------:|------------:|------------:|------------:| | BM25 | 61.35 | 66.89 | **96.38** | 74.87 | | BM25 (lemma) | 71.49 | 75.33 | 94.57 | 80.46 | | MiniLM-L12-v2 | 37.24 | 71.67 | 78.97 | 62.62 | | LaBSE | 46.23 | 67.11 | 81.34 | 64.89 | | mContriever-Base | 78.66 | 84.44 | 95.82 | 86.31 | | E5-Base | 86.61 | 91.89 | 96.24 | 91.58 | | ST-DistilRoBERTa | 48.43 | 84.89 | 88.02 | 73.78 | | ST-MPNet | 56.80 | 86.00 | 87.19 | 76.66 | | HerBERT-QA | 75.84 | 85.78 | 91.09 | 84.23 | | **SilverRetriever** | **87.24** | **94.56** | 95.54 | **92.45** | ### NDCG@10 | **Model** | **PolQA** | **Allegro FAQ** | **Legal Questions** | **Average** | |:-----------------------|-------------:|-------------:|-------------:|-------------:| | BM25 | 24.51 | 48.71 | **82.21** | 51.81 | | BM25 (lemma) | 31.97 | 55.70 | 78.65 | 55.44 | | MiniLM-L12-v2 | 11.93 | 51.25 | 54.44 | 39.21 | | LaBSE | 15.53 | 46.71 | 56.16 | 39.47 | | mContriever-Base | 36.30 | 67.38 | 77.42 | 60.37 | | E5-Base | **46.08** | 75.90 | 77.69 | 66.56 | | ST-DistilRoBERTa | 16.73 | 64.39 | 63.76 | 48.29 | | ST-MPNet | 21.55 | 65.44 | 62.99 | 49.99 | | HerBERT-QA | 32.52 | 63.58 | 66.99 | 54.36 | | **SilverRetriever** | 43.40 | **79.66** | 77.10 | **66.72** | ## Usage ### Preparing inputs The model was trained on question-passage pairs and works best when the input is the same format as that used during training: - We added the phrase `Pytanie:` to the beginning of the question. - The training passages consisted of `title` and `text` concatenated with the special token ``. Even if your passages don't have a `title`, it is still beneficial to prefix a passage with the `` token. - Although we used the dot product during training, the model usually works better with the cosine distance. ### Inference with Sentence-Transformers Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: ``` pip install -U sentence-transformers ``` Then you can use the model like this: ```python from sentence_transformers import SentenceTransformer sentences = [ "Pytanie: W jakim mieście urodził się Zbigniew Herbert?", "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.", ] model = SentenceTransformer('ipipan/silver-retriever-base-v1') embeddings = model.encode(sentences) print(embeddings) ``` ### Inference with HuggingFace Transformers Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings. ```python from transformers import AutoTokenizer, AutoModel import torch def cls_pooling(model_output, attention_mask): return model_output[0][:,0] # Sentences we want sentence embeddings for sentences = [ "Pytanie: W jakim mieście urodził się Zbigniew Herbert?", "Zbigniew HerbertZbigniew Bolesław Ryszard Herbert (ur. 29 października 1924 we Lwowie, zm. 28 lipca 1998 w Warszawie) – polski poeta, eseista i dramaturg.", ] # Load model from HuggingFace Hub tokenizer = AutoTokenizer.from_pretrained('ipipan/silver-retriever-base-v1') model = AutoModel.from_pretrained('ipipan/silver-retriever-base-v1') # Tokenize sentences encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') # Compute token embeddings with torch.no_grad(): model_output = model(**encoded_input) # Perform pooling. In this case, cls pooling. sentence_embeddings = cls_pooling(model_output, encoded_input['attention_mask']) print("Sentence embeddings:") print(sentence_embeddings) ``` ## Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False}) ) ``` ## Additional Information ### Model Creators The model was created by Piotr Rybak from the [Institute of Computer Science, Polish Academy of Sciences](http://zil.ipipan.waw.pl/). This work was supported by the European Regional Development Fund as a part of 2014–2020 Smart Growth Operational Programme, CLARIN — Common Language Resources and Technology Infrastructure, project no. POIR.04.02.00-00C002/19. ### Licensing Information CC BY-SA 4.0 ### Citation Information [More Information Needed]