--- pipeline_tag: sentence-similarity tags: - sentence-transformers - feature-extraction - sentence-similarity - mteb model-index: - name: SGPT-125M-weightedmean-msmarco-specb-bitfit results: - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en) metrics: - type: accuracy value: 0.6123880597014926 - type: ap value: 0.25854431650388643 - type: f1 value: 0.557518627628186 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (de) metrics: - type: accuracy value: 0.5688436830835117 - type: ap value: 0.7267279104379771 - type: f1 value: 0.5444984024378641 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en-ext) metrics: - type: accuracy value: 0.5827586206896551 - type: ap value: 0.14067357642500386 - type: f1 value: 0.4817231851869133 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (ja) metrics: - type: accuracy value: 0.5464668094218414 - type: ap value: 0.11776694555054965 - type: f1 value: 0.44526622834078766 - task: type: Classification dataset: type: mteb/amazon_polarity name: MTEB AmazonPolarityClassification metrics: - type: accuracy value: 0.65401225 - type: ap value: 0.6022809958678552 - type: f1 value: 0.650251824898292 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (en) metrics: - type: accuracy value: 0.31165999999999994 - type: f1 value: 0.30908870050167436 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (de) metrics: - type: accuracy value: 0.2479 - type: f1 value: 0.245833598854121 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (es) metrics: - type: accuracy value: 0.26643999999999995 - type: f1 value: 0.2639012792213563 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (fr) metrics: - type: accuracy value: 0.26386000000000004 - type: f1 value: 0.2627686779145487 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (ja) metrics: - type: accuracy value: 0.22078000000000003 - type: f1 value: 0.21797960290226842 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (zh) metrics: - type: accuracy value: 0.24274 - type: f1 value: 0.23887054434822627 - task: type: Retrieval dataset: type: arguana name: MTEB ArguAna metrics: - type: map_at_1 value: 0.22404 - type: map_at_10 value: 0.36845 - type: map_at_100 value: 0.37945 - type: map_at_1000 value: 0.37966 - type: map_at_3 value: 0.3178 - type: map_at_5 value: 0.34608 - type: mrr_at_1 value: 0.22902 - type: mrr_at_10 value: 0.37034 - type: mrr_at_100 value: 0.38134 - type: mrr_at_1000 value: 0.38155 - type: mrr_at_3 value: 0.31935 - type: mrr_at_5 value: 0.34812 - type: ndcg_at_1 value: 0.22404 - type: ndcg_at_10 value: 0.45425 - type: ndcg_at_100 value: 0.50354 - type: ndcg_at_1000 value: 0.50874 - type: ndcg_at_3 value: 0.3497 - type: ndcg_at_5 value: 0.40081 - type: precision_at_1 value: 0.22404 - type: precision_at_10 value: 0.07304 - type: precision_at_100 value: 0.00951 - type: precision_at_1000 value: 0.00099 - type: precision_at_3 value: 0.14746 - type: precision_at_5 value: 0.11337 - type: recall_at_1 value: 0.22404 - type: recall_at_10 value: 0.73044 - type: recall_at_100 value: 0.95092 - type: recall_at_1000 value: 0.99075 - type: recall_at_3 value: 0.44239 - type: recall_at_5 value: 0.56686 - task: type: Clustering dataset: type: mteb/arxiv-clustering-p2p name: MTEB ArxivClusteringP2P metrics: - type: v_measure value: 0.3970858340673288 - task: type: Clustering dataset: type: mteb/arxiv-clustering-s2s name: MTEB ArxivClusteringS2S metrics: - type: v_measure value: 0.2824284771372105 - task: type: Reranking dataset: type: mteb/askubuntudupquestions-reranking name: MTEB AskUbuntuDupQuestions metrics: - type: map value: 0.5583700395192394 - type: mrr value: 0.7038913072154069 - task: type: STS dataset: type: mteb/biosses-sts name: MTEB BIOSSES metrics: - type: cos_sim_pearson value: 0.7925366801756224 - type: cos_sim_spearman value: 0.7520954502580506 - type: euclidean_pearson value: 0.7879900722991617 - type: euclidean_spearman value: 0.7779996549607588 - type: manhattan_pearson value: 0.7818408109480399 - type: manhattan_spearman value: 0.7685958262303105 - task: type: Classification dataset: type: mteb/banking77 name: MTEB Banking77Classification metrics: - type: accuracy value: 0.7770454545454545 - type: f1 value: 0.7769290001138031 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-p2p name: MTEB BiorxivClusteringP2P metrics: - type: v_measure value: 0.33632603955439844 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-s2s name: MTEB BiorxivClusteringS2S metrics: - type: v_measure value: 0.27038042665369927 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackAndroidRetrieval metrics: - type: map_at_1 value: 0.22139 - type: map_at_10 value: 0.28839 - type: map_at_100 value: 0.30023 - type: map_at_1000 value: 0.30153 - type: map_at_3 value: 0.26521 - type: map_at_5 value: 0.27775 - type: mrr_at_1 value: 0.26466 - type: mrr_at_10 value: 0.33495 - type: mrr_at_100 value: 0.34417 - type: mrr_at_1000 value: 0.34485 - type: mrr_at_3 value: 0.31402 - type: mrr_at_5 value: 0.32496 - type: ndcg_at_1 value: 0.26466 - type: ndcg_at_10 value: 0.33372 - type: ndcg_at_100 value: 0.387 - type: ndcg_at_1000 value: 0.41696 - type: ndcg_at_3 value: 0.29443 - type: ndcg_at_5 value: 0.31121 - type: precision_at_1 value: 0.26466 - type: precision_at_10 value: 0.06037 - type: precision_at_100 value: 0.01067 - type: precision_at_1000 value: 0.00162 - type: precision_at_3 value: 0.13782 - type: precision_at_5 value: 0.09757 - type: recall_at_1 value: 0.22139 - type: recall_at_10 value: 0.4239 - type: recall_at_100 value: 0.65427 - type: recall_at_1000 value: 0.86049 - type: recall_at_3 value: 0.31127 - type: recall_at_5 value: 0.35718 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackEnglishRetrieval metrics: - type: map_at_1 value: 0.20652 - type: map_at_10 value: 0.27558 - type: map_at_100 value: 0.28473 - type: map_at_1000 value: 0.28577 - type: map_at_3 value: 0.25402 - type: map_at_5 value: 0.2668 - type: mrr_at_1 value: 0.25223 - type: mrr_at_10 value: 0.31966 - type: mrr_at_100 value: 0.32664 - type: mrr_at_1000 value: 0.32724 - type: mrr_at_3 value: 0.30074 - type: mrr_at_5 value: 0.31249 - type: ndcg_at_1 value: 0.25223 - type: ndcg_at_10 value: 0.31694 - type: ndcg_at_100 value: 0.35662 - type: ndcg_at_1000 value: 0.38092 - type: ndcg_at_3 value: 0.28294 - type: ndcg_at_5 value: 0.30049 - type: precision_at_1 value: 0.25223 - type: precision_at_10 value: 0.05777 - type: precision_at_100 value: 0.00973 - type: precision_at_1000 value: 0.0014 - type: precision_at_3 value: 0.13397 - type: precision_at_5 value: 0.09605 - type: recall_at_1 value: 0.20652 - type: recall_at_10 value: 0.39368 - type: recall_at_100 value: 0.56485 - type: recall_at_1000 value: 0.73292 - type: recall_at_3 value: 0.2983 - type: recall_at_5 value: 0.3443 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGamingRetrieval metrics: - type: map_at_1 value: 0.2518 - type: map_at_10 value: 0.34579 - type: map_at_100 value: 0.3559 - type: map_at_1000 value: 0.3568 - type: map_at_3 value: 0.31736 - type: map_at_5 value: 0.33479 - type: mrr_at_1 value: 0.29467 - type: mrr_at_10 value: 0.37967 - type: mrr_at_100 value: 0.388 - type: mrr_at_1000 value: 0.38858 - type: mrr_at_3 value: 0.35465 - type: mrr_at_5 value: 0.37057 - type: ndcg_at_1 value: 0.29467 - type: ndcg_at_10 value: 0.39796 - type: ndcg_at_100 value: 0.44531 - type: ndcg_at_1000 value: 0.46666 - type: ndcg_at_3 value: 0.34676 - type: ndcg_at_5 value: 0.37468 - type: precision_at_1 value: 0.29467 - type: precision_at_10 value: 0.06602 - type: precision_at_100 value: 0.0099 - type: precision_at_1000 value: 0.00124 - type: precision_at_3 value: 0.15569 - type: precision_at_5 value: 0.11172 - type: recall_at_1 value: 0.2518 - type: recall_at_10 value: 0.52269 - type: recall_at_100 value: 0.73574 - type: recall_at_1000 value: 0.89141 - type: recall_at_3 value: 0.38522 - type: recall_at_5 value: 0.45323 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGisRetrieval metrics: - type: map_at_1 value: 0.16303 - type: map_at_10 value: 0.21629 - type: map_at_100 value: 0.22388 - type: map_at_1000 value: 0.22489 - type: map_at_3 value: 0.19608 - type: map_at_5 value: 0.20774 - type: mrr_at_1 value: 0.1774 - type: mrr_at_10 value: 0.23214 - type: mrr_at_100 value: 0.2397 - type: mrr_at_1000 value: 0.24054 - type: mrr_at_3 value: 0.21243 - type: mrr_at_5 value: 0.22322 - type: ndcg_at_1 value: 0.1774 - type: ndcg_at_10 value: 0.25113 - type: ndcg_at_100 value: 0.29288 - type: ndcg_at_1000 value: 0.32204 - type: ndcg_at_3 value: 0.21111 - type: ndcg_at_5 value: 0.23062 - type: precision_at_1 value: 0.1774 - type: precision_at_10 value: 0.03955 - type: precision_at_100 value: 0.00644 - type: precision_at_1000 value: 0.00093 - type: precision_at_3 value: 0.08851 - type: precision_at_5 value: 0.06418 - type: recall_at_1 value: 0.16303 - type: recall_at_10 value: 0.34487 - type: recall_at_100 value: 0.54414 - type: recall_at_1000 value: 0.77158 - type: recall_at_3 value: 0.23733 - type: recall_at_5 value: 0.28381 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackMathematicaRetrieval metrics: - type: map_at_1 value: 0.10133 - type: map_at_10 value: 0.15666 - type: map_at_100 value: 0.16592 - type: map_at_1000 value: 0.16734 - type: map_at_3 value: 0.13625 - type: map_at_5 value: 0.14721 - type: mrr_at_1 value: 0.12562 - type: mrr_at_10 value: 0.18487 - type: mrr_at_100 value: 0.19391 - type: mrr_at_1000 value: 0.19487 - type: mrr_at_3 value: 0.16418 - type: mrr_at_5 value: 0.176 - type: ndcg_at_1 value: 0.12562 - type: ndcg_at_10 value: 0.1943 - type: ndcg_at_100 value: 0.24546 - type: ndcg_at_1000 value: 0.28193 - type: ndcg_at_3 value: 0.1551 - type: ndcg_at_5 value: 0.17322 - type: precision_at_1 value: 0.12562 - type: precision_at_10 value: 0.03794 - type: precision_at_100 value: 0.0074 - type: precision_at_1000 value: 0.00122 - type: precision_at_3 value: 0.07546 - type: precision_at_5 value: 0.05721 - type: recall_at_1 value: 0.10133 - type: recall_at_10 value: 0.28262 - type: recall_at_100 value: 0.51743 - type: recall_at_1000 value: 0.78075 - type: recall_at_3 value: 0.17634 - type: recall_at_5 value: 0.22129 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackPhysicsRetrieval metrics: - type: map_at_1 value: 0.19992 - type: map_at_10 value: 0.27347 - type: map_at_100 value: 0.28582 - type: map_at_1000 value: 0.28716 - type: map_at_3 value: 0.24907 - type: map_at_5 value: 0.261 - type: mrr_at_1 value: 0.23773 - type: mrr_at_10 value: 0.31647 - type: mrr_at_100 value: 0.32639 - type: mrr_at_1000 value: 0.32706 - type: mrr_at_3 value: 0.29195 - type: mrr_at_5 value: 0.30484 - type: ndcg_at_1 value: 0.23773 - type: ndcg_at_10 value: 0.32322 - type: ndcg_at_100 value: 0.37996 - type: ndcg_at_1000 value: 0.40819 - type: ndcg_at_3 value: 0.27876 - type: ndcg_at_5 value: 0.29664 - type: precision_at_1 value: 0.23773 - type: precision_at_10 value: 0.05977 - type: precision_at_100 value: 0.01055 - type: precision_at_1000 value: 0.0015 - type: precision_at_3 value: 0.13122 - type: precision_at_5 value: 0.09451 - type: recall_at_1 value: 0.19992 - type: recall_at_10 value: 0.43106 - type: recall_at_100 value: 0.67264 - type: recall_at_1000 value: 0.86386 - type: recall_at_3 value: 0.30392 - type: recall_at_5 value: 0.34911 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackProgrammersRetrieval metrics: - type: map_at_1 value: 0.17896 - type: map_at_10 value: 0.24644 - type: map_at_100 value: 0.2579 - type: map_at_1000 value: 0.25914 - type: map_at_3 value: 0.22694 - type: map_at_5 value: 0.2369 - type: mrr_at_1 value: 0.21347 - type: mrr_at_10 value: 0.28594 - type: mrr_at_100 value: 0.29544 - type: mrr_at_1000 value: 0.29621 - type: mrr_at_3 value: 0.26807 - type: mrr_at_5 value: 0.27669 - type: ndcg_at_1 value: 0.21347 - type: ndcg_at_10 value: 0.28833 - type: ndcg_at_100 value: 0.34272 - type: ndcg_at_1000 value: 0.37355 - type: ndcg_at_3 value: 0.25373 - type: ndcg_at_5 value: 0.26756 - type: precision_at_1 value: 0.21347 - type: precision_at_10 value: 0.05217 - type: precision_at_100 value: 0.00954 - type: precision_at_1000 value: 0.00139 - type: precision_at_3 value: 0.11948 - type: precision_at_5 value: 0.08425 - type: recall_at_1 value: 0.17896 - type: recall_at_10 value: 0.37291 - type: recall_at_100 value: 0.61138 - type: recall_at_1000 value: 0.83212 - type: recall_at_3 value: 0.27706 - type: recall_at_5 value: 0.31234 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackRetrieval metrics: - type: map_at_1 value: 0.17195166666666667 - type: map_at_10 value: 0.23329083333333334 - type: map_at_100 value: 0.2430308333333333 - type: map_at_1000 value: 0.24422416666666666 - type: map_at_3 value: 0.21327416666666665 - type: map_at_5 value: 0.22419999999999998 - type: mrr_at_1 value: 0.19999916666666667 - type: mrr_at_10 value: 0.26390166666666665 - type: mrr_at_100 value: 0.27231 - type: mrr_at_1000 value: 0.27308333333333334 - type: mrr_at_3 value: 0.244675 - type: mrr_at_5 value: 0.25541083333333336 - type: ndcg_at_1 value: 0.19999916666666667 - type: ndcg_at_10 value: 0.27248666666666665 - type: ndcg_at_100 value: 0.3200258333333334 - type: ndcg_at_1000 value: 0.34946499999999997 - type: ndcg_at_3 value: 0.2358566666666667 - type: ndcg_at_5 value: 0.2526341666666666 - type: precision_at_1 value: 0.19999916666666667 - type: precision_at_10 value: 0.04772166666666666 - type: precision_at_100 value: 0.00847 - type: precision_at_1000 value: 0.0012741666666666667 - type: precision_at_3 value: 0.10756166666666668 - type: precision_at_5 value: 0.07725416666666667 - type: recall_at_1 value: 0.17195166666666667 - type: recall_at_10 value: 0.35990833333333344 - type: recall_at_100 value: 0.57468 - type: recall_at_1000 value: 0.7882366666666667 - type: recall_at_3 value: 0.25898499999999997 - type: recall_at_5 value: 0.30084333333333335 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackStatsRetrieval metrics: - type: map_at_1 value: 0.16779 - type: map_at_10 value: 0.21557 - type: map_at_100 value: 0.22338 - type: map_at_1000 value: 0.22421 - type: map_at_3 value: 0.19939 - type: map_at_5 value: 0.20903 - type: mrr_at_1 value: 0.18405 - type: mrr_at_10 value: 0.23435 - type: mrr_at_100 value: 0.24179 - type: mrr_at_1000 value: 0.2425 - type: mrr_at_3 value: 0.21907 - type: mrr_at_5 value: 0.22781 - type: ndcg_at_1 value: 0.18405 - type: ndcg_at_10 value: 0.24515 - type: ndcg_at_100 value: 0.28721 - type: ndcg_at_1000 value: 0.3126 - type: ndcg_at_3 value: 0.21508 - type: ndcg_at_5 value: 0.2301 - type: precision_at_1 value: 0.18405 - type: precision_at_10 value: 0.03834 - type: precision_at_100 value: 0.00641 - type: precision_at_1000 value: 0.00093 - type: precision_at_3 value: 0.09151 - type: precision_at_5 value: 0.06503 - type: recall_at_1 value: 0.16779 - type: recall_at_10 value: 0.3173 - type: recall_at_100 value: 0.51673 - type: recall_at_1000 value: 0.71176 - type: recall_at_3 value: 0.23518 - type: recall_at_5 value: 0.27231 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackTexRetrieval metrics: - type: map_at_1 value: 0.09279 - type: map_at_10 value: 0.13822 - type: map_at_100 value: 0.14533 - type: map_at_1000 value: 0.1465 - type: map_at_3 value: 0.12396 - type: map_at_5 value: 0.13214 - type: mrr_at_1 value: 0.11149 - type: mrr_at_10 value: 0.16139 - type: mrr_at_100 value: 0.16872 - type: mrr_at_1000 value: 0.16964 - type: mrr_at_3 value: 0.14613 - type: mrr_at_5 value: 0.15486 - type: ndcg_at_1 value: 0.11149 - type: ndcg_at_10 value: 0.1682 - type: ndcg_at_100 value: 0.2073 - type: ndcg_at_1000 value: 0.23894 - type: ndcg_at_3 value: 0.1411 - type: ndcg_at_5 value: 0.15404 - type: precision_at_1 value: 0.11149 - type: precision_at_10 value: 0.03063 - type: precision_at_100 value: 0.00587 - type: precision_at_1000 value: 0.001 - type: precision_at_3 value: 0.06699 - type: precision_at_5 value: 0.04928 - type: recall_at_1 value: 0.09279 - type: recall_at_10 value: 0.23745 - type: recall_at_100 value: 0.41873 - type: recall_at_1000 value: 0.64982 - type: recall_at_3 value: 0.16152 - type: recall_at_5 value: 0.19409 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackUnixRetrieval metrics: - type: map_at_1 value: 0.1636 - type: map_at_10 value: 0.21927 - type: map_at_100 value: 0.22889 - type: map_at_1000 value: 0.22994 - type: map_at_3 value: 0.20433 - type: map_at_5 value: 0.21337 - type: mrr_at_1 value: 0.1875 - type: mrr_at_10 value: 0.24859 - type: mrr_at_100 value: 0.25747 - type: mrr_at_1000 value: 0.25829 - type: mrr_at_3 value: 0.23383 - type: mrr_at_5 value: 0.24297 - type: ndcg_at_1 value: 0.1875 - type: ndcg_at_10 value: 0.25372 - type: ndcg_at_100 value: 0.30343 - type: ndcg_at_1000 value: 0.33286 - type: ndcg_at_3 value: 0.22627 - type: ndcg_at_5 value: 0.2404 - type: precision_at_1 value: 0.1875 - type: precision_at_10 value: 0.04142 - type: precision_at_100 value: 0.00738 - type: precision_at_1000 value: 0.00111 - type: precision_at_3 value: 0.10261 - type: precision_at_5 value: 0.07164 - type: recall_at_1 value: 0.1636 - type: recall_at_10 value: 0.32949 - type: recall_at_100 value: 0.55552 - type: recall_at_1000 value: 0.77099 - type: recall_at_3 value: 0.25538 - type: recall_at_5 value: 0.29008 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWebmastersRetrieval metrics: - type: map_at_1 value: 0.1739 - type: map_at_10 value: 0.23058 - type: map_at_100 value: 0.24445 - type: map_at_1000 value: 0.24638 - type: map_at_3 value: 0.21037 - type: map_at_5 value: 0.21966 - type: mrr_at_1 value: 0.1996 - type: mrr_at_10 value: 0.26301 - type: mrr_at_100 value: 0.27297 - type: mrr_at_1000 value: 0.27375 - type: mrr_at_3 value: 0.24341 - type: mrr_at_5 value: 0.25339 - type: ndcg_at_1 value: 0.1996 - type: ndcg_at_10 value: 0.27249 - type: ndcg_at_100 value: 0.32997 - type: ndcg_at_1000 value: 0.36359 - type: ndcg_at_3 value: 0.23519 - type: ndcg_at_5 value: 0.24915 - type: precision_at_1 value: 0.1996 - type: precision_at_10 value: 0.05356 - type: precision_at_100 value: 0.01198 - type: precision_at_1000 value: 0.00204 - type: precision_at_3 value: 0.10738 - type: precision_at_5 value: 0.07905 - type: recall_at_1 value: 0.1739 - type: recall_at_10 value: 0.35255 - type: recall_at_100 value: 0.61351 - type: recall_at_1000 value: 0.84395 - type: recall_at_3 value: 0.25194 - type: recall_at_5 value: 0.28546 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWordpressRetrieval metrics: - type: map_at_1 value: 0.14239 - type: map_at_10 value: 0.19323 - type: map_at_100 value: 0.19994 - type: map_at_1000 value: 0.20103 - type: map_at_3 value: 0.17631 - type: map_at_5 value: 0.18401 - type: mrr_at_1 value: 0.15157 - type: mrr_at_10 value: 0.20578 - type: mrr_at_100 value: 0.21252 - type: mrr_at_1000 value: 0.21347 - type: mrr_at_3 value: 0.18762 - type: mrr_at_5 value: 0.19713 - type: ndcg_at_1 value: 0.15157 - type: ndcg_at_10 value: 0.22468 - type: ndcg_at_100 value: 0.26245 - type: ndcg_at_1000 value: 0.29534 - type: ndcg_at_3 value: 0.18981 - type: ndcg_at_5 value: 0.2035 - type: precision_at_1 value: 0.15157 - type: precision_at_10 value: 0.03512 - type: precision_at_100 value: 0.00577 - type: precision_at_1000 value: 0.00091 - type: precision_at_3 value: 0.0801 - type: precision_at_5 value: 0.05656 - type: recall_at_1 value: 0.14239 - type: recall_at_10 value: 0.31038 - type: recall_at_100 value: 0.49122 - type: recall_at_1000 value: 0.74919 - type: recall_at_3 value: 0.21436 - type: recall_at_5 value: 0.24692 - task: type: Retrieval dataset: type: climate-fever name: MTEB ClimateFEVER metrics: - type: map_at_1 value: 0.08828 - type: map_at_10 value: 0.14982 - type: map_at_100 value: 0.16495 - type: map_at_1000 value: 0.16658 - type: map_at_3 value: 0.12366 - type: map_at_5 value: 0.13655 - type: mrr_at_1 value: 0.19088 - type: mrr_at_10 value: 0.2929 - type: mrr_at_100 value: 0.30291 - type: mrr_at_1000 value: 0.30342 - type: mrr_at_3 value: 0.25907 - type: mrr_at_5 value: 0.27841 - type: ndcg_at_1 value: 0.19088 - type: ndcg_at_10 value: 0.21858 - type: ndcg_at_100 value: 0.28324 - type: ndcg_at_1000 value: 0.31561 - type: ndcg_at_3 value: 0.17175 - type: ndcg_at_5 value: 0.18869 - type: precision_at_1 value: 0.19088 - type: precision_at_10 value: 0.06919 - type: precision_at_100 value: 0.01376 - type: precision_at_1000 value: 0.00197 - type: precision_at_3 value: 0.12704 - type: precision_at_5 value: 0.09993 - type: recall_at_1 value: 0.08828 - type: recall_at_10 value: 0.27381 - type: recall_at_100 value: 0.5 - type: recall_at_1000 value: 0.68355 - type: recall_at_3 value: 0.16118 - type: recall_at_5 value: 0.20587 - task: type: Retrieval dataset: type: dbpedia-entity name: MTEB DBPedia metrics: - type: map_at_1 value: 0.05586 - type: map_at_10 value: 0.1004 - type: map_at_100 value: 0.1255 - type: map_at_1000 value: 0.13124 - type: map_at_3 value: 0.0775 - type: map_at_5 value: 0.08836 - type: mrr_at_1 value: 0.4225 - type: mrr_at_10 value: 0.51206 - type: mrr_at_100 value: 0.51818 - type: mrr_at_1000 value: 0.51855 - type: mrr_at_3 value: 0.48875 - type: mrr_at_5 value: 0.50488 - type: ndcg_at_1 value: 0.3225 - type: ndcg_at_10 value: 0.22718 - type: ndcg_at_100 value: 0.24359 - type: ndcg_at_1000 value: 0.29232 - type: ndcg_at_3 value: 0.25974 - type: ndcg_at_5 value: 0.24292 - type: precision_at_1 value: 0.4225 - type: precision_at_10 value: 0.1775 - type: precision_at_100 value: 0.05032 - type: precision_at_1000 value: 0.01117 - type: precision_at_3 value: 0.28833 - type: precision_at_5 value: 0.2425 - type: recall_at_1 value: 0.05586 - type: recall_at_10 value: 0.1416 - type: recall_at_100 value: 0.28051 - type: recall_at_1000 value: 0.45157 - type: recall_at_3 value: 0.08758 - type: recall_at_5 value: 0.10976 - task: type: Classification dataset: type: mteb/emotion name: MTEB EmotionClassification metrics: - type: accuracy value: 0.39075000000000004 - type: f1 value: 0.3501420354708222 - task: type: Retrieval dataset: type: fever name: MTEB FEVER metrics: - type: map_at_1 value: 0.4352 - type: map_at_10 value: 0.54368 - type: map_at_100 value: 0.54918 - type: map_at_1000 value: 0.54942 - type: map_at_3 value: 0.51712 - type: map_at_5 value: 0.53336 - type: mrr_at_1 value: 0.46955 - type: mrr_at_10 value: 0.58219 - type: mrr_at_100 value: 0.58735 - type: mrr_at_1000 value: 0.58753 - type: mrr_at_3 value: 0.55518 - type: mrr_at_5 value: 0.57191 - type: ndcg_at_1 value: 0.46955 - type: ndcg_at_10 value: 0.6045 - type: ndcg_at_100 value: 0.63047 - type: ndcg_at_1000 value: 0.63713 - type: ndcg_at_3 value: 0.55233 - type: ndcg_at_5 value: 0.58072 - type: precision_at_1 value: 0.46955 - type: precision_at_10 value: 0.08267 - type: precision_at_100 value: 0.00962 - type: precision_at_1000 value: 0.00103 - type: precision_at_3 value: 0.22327 - type: precision_at_5 value: 0.14941 - type: recall_at_1 value: 0.4352 - type: recall_at_10 value: 0.75632 - type: recall_at_100 value: 0.87416 - type: recall_at_1000 value: 0.92557 - type: recall_at_3 value: 0.61597 - type: recall_at_5 value: 0.68518 - task: type: Retrieval dataset: type: fiqa name: MTEB FiQA2018 metrics: - type: map_at_1 value: 0.09549 - type: map_at_10 value: 0.15762 - type: map_at_100 value: 0.17142 - type: map_at_1000 value: 0.17329 - type: map_at_3 value: 0.13575 - type: map_at_5 value: 0.14754 - type: mrr_at_1 value: 0.19753 - type: mrr_at_10 value: 0.26568 - type: mrr_at_100 value: 0.27606 - type: mrr_at_1000 value: 0.2768 - type: mrr_at_3 value: 0.24203 - type: mrr_at_5 value: 0.25669 - type: ndcg_at_1 value: 0.19753 - type: ndcg_at_10 value: 0.21118 - type: ndcg_at_100 value: 0.27308 - type: ndcg_at_1000 value: 0.31304 - type: ndcg_at_3 value: 0.18319 - type: ndcg_at_5 value: 0.19414 - type: precision_at_1 value: 0.19753 - type: precision_at_10 value: 0.0608 - type: precision_at_100 value: 0.01204 - type: precision_at_1000 value: 0.00192 - type: precision_at_3 value: 0.12191 - type: precision_at_5 value: 0.09383 - type: recall_at_1 value: 0.09549 - type: recall_at_10 value: 0.26131 - type: recall_at_100 value: 0.50545 - type: recall_at_1000 value: 0.74968 - type: recall_at_3 value: 0.16951 - type: recall_at_5 value: 0.2095 - task: type: Retrieval dataset: type: hotpotqa name: MTEB HotpotQA metrics: - type: map_at_1 value: 0.25544 - type: map_at_10 value: 0.3262 - type: map_at_100 value: 0.33275 - type: map_at_1000 value: 0.33344 - type: map_at_3 value: 0.30851 - type: map_at_5 value: 0.31869 - type: mrr_at_1 value: 0.51087 - type: mrr_at_10 value: 0.57704 - type: mrr_at_100 value: 0.58175 - type: mrr_at_1000 value: 0.58207 - type: mrr_at_3 value: 0.56106 - type: mrr_at_5 value: 0.57074 - type: ndcg_at_1 value: 0.51087 - type: ndcg_at_10 value: 0.40876 - type: ndcg_at_100 value: 0.43762 - type: ndcg_at_1000 value: 0.45423 - type: ndcg_at_3 value: 0.3765 - type: ndcg_at_5 value: 0.39305 - type: precision_at_1 value: 0.51087 - type: precision_at_10 value: 0.08304 - type: precision_at_100 value: 0.01059 - type: precision_at_1000 value: 0.00128 - type: precision_at_3 value: 0.22876 - type: precision_at_5 value: 0.15033 - type: recall_at_1 value: 0.25544 - type: recall_at_10 value: 0.41519 - type: recall_at_100 value: 0.52957 - type: recall_at_1000 value: 0.64132 - type: recall_at_3 value: 0.34315 - type: recall_at_5 value: 0.37583 - task: type: Classification dataset: type: mteb/imdb name: MTEB ImdbClassification metrics: - type: accuracy value: 0.586696 - type: ap value: 0.553644880984279 - type: f1 value: 0.5807942097405652 - task: type: Retrieval dataset: type: msmarco name: MTEB MSMARCO metrics: - type: map_at_1 value: 0.14442 - type: map_at_10 value: 0.22932 - type: map_at_100 value: 0.24132 - type: map_at_1000 value: 0.24213 - type: map_at_3 value: 0.20002 - type: map_at_5 value: 0.21636 - type: mrr_at_1 value: 0.14842 - type: mrr_at_10 value: 0.23416 - type: mrr_at_100 value: 0.24594 - type: mrr_at_1000 value: 0.24669 - type: mrr_at_3 value: 0.20494 - type: mrr_at_5 value: 0.2214 - type: ndcg_at_1 value: 0.14842 - type: ndcg_at_10 value: 0.27975 - type: ndcg_at_100 value: 0.34143 - type: ndcg_at_1000 value: 0.3637 - type: ndcg_at_3 value: 0.21944 - type: ndcg_at_5 value: 0.24881 - type: precision_at_1 value: 0.14842 - type: precision_at_10 value: 0.04537 - type: precision_at_100 value: 0.00767 - type: precision_at_1000 value: 0.00096 - type: precision_at_3 value: 0.09322 - type: precision_at_5 value: 0.07074 - type: recall_at_1 value: 0.14442 - type: recall_at_10 value: 0.43557 - type: recall_at_100 value: 0.72904 - type: recall_at_1000 value: 0.90407 - type: recall_at_3 value: 0.27088 - type: recall_at_5 value: 0.34144 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (en) metrics: - type: accuracy value: 0.8695622435020519 - type: f1 value: 0.8658363130708494 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (de) metrics: - type: accuracy value: 0.6273034657650043 - type: f1 value: 0.6078623915840713 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (es) metrics: - type: accuracy value: 0.6754503002001334 - type: f1 value: 0.6534879794116112 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (fr) metrics: - type: accuracy value: 0.653523332289383 - type: f1 value: 0.6299400188244665 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (hi) metrics: - type: accuracy value: 0.45371100752958055 - type: f1 value: 0.4426285860740745 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (th) metrics: - type: accuracy value: 0.5527667269439421 - type: f1 value: 0.5328388179869588 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (en) metrics: - type: accuracy value: 0.6225262197902417 - type: f1 value: 0.43440840371488526 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (de) metrics: - type: accuracy value: 0.4956043956043956 - type: f1 value: 0.32863336734985976 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (es) metrics: - type: accuracy value: 0.4993995997331555 - type: f1 value: 0.34726671876888127 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (fr) metrics: - type: accuracy value: 0.46329470717193855 - type: f1 value: 0.3232527361598279 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (hi) metrics: - type: accuracy value: 0.3220867694514163 - type: f1 value: 0.21321851228151392 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (th) metrics: - type: accuracy value: 0.43627486437613017 - type: f1 value: 0.27048729223475076 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (af) metrics: - type: accuracy value: 0.4054808338937458 - type: f1 value: 0.39490307545239717 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (am) metrics: - type: accuracy value: 0.2418291862811029 - type: f1 value: 0.23437620034727474 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ar) metrics: - type: accuracy value: 0.30134498991257563 - type: f1 value: 0.28787175191531283 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (az) metrics: - type: accuracy value: 0.35884330867518494 - type: f1 value: 0.36264500398782124 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (bn) metrics: - type: accuracy value: 0.2917283120376597 - type: f1 value: 0.278101616531901 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (cy) metrics: - type: accuracy value: 0.41788836583725625 - type: f1 value: 0.39714131810548015 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (da) metrics: - type: accuracy value: 0.44176193678547404 - type: f1 value: 0.4219249982655229 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (de) metrics: - type: accuracy value: 0.4207464694014795 - type: f1 value: 0.39441882591831623 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (el) metrics: - type: accuracy value: 0.362542030934768 - type: f1 value: 0.3446592715936761 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (en) metrics: - type: accuracy value: 0.6140887693342301 - type: f1 value: 0.5979854802683996 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (es) metrics: - type: accuracy value: 0.42679892400806996 - type: f1 value: 0.4204801248338172 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fa) metrics: - type: accuracy value: 0.3559179556153329 - type: f1 value: 0.34045862930486165 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fi) metrics: - type: accuracy value: 0.40036987222595827 - type: f1 value: 0.3811770343936278 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fr) metrics: - type: accuracy value: 0.4343981170141224 - type: f1 value: 0.42708438898786494 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (he) metrics: - type: accuracy value: 0.3159381304640215 - type: f1 value: 0.2998550522450782 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hi) metrics: - type: accuracy value: 0.27044384667114996 - type: f1 value: 0.27313059184832666 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hu) metrics: - type: accuracy value: 0.38453261600538 - type: f1 value: 0.37309189326110437 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hy) metrics: - type: accuracy value: 0.2797915265635508 - type: f1 value: 0.27430939684346445 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (id) metrics: - type: accuracy value: 0.4397108271687963 - type: f1 value: 0.43405857056887615 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (is) metrics: - type: accuracy value: 0.40302622730329524 - type: f1 value: 0.39108052180520747 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (it) metrics: - type: accuracy value: 0.45474108944182917 - type: f1 value: 0.4585950328241134 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ja) metrics: - type: accuracy value: 0.4560860793544048 - type: f1 value: 0.4394920708216737 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (jv) metrics: - type: accuracy value: 0.386684599865501 - type: f1 value: 0.37699003401885905 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ka) metrics: - type: accuracy value: 0.25652320107599197 - type: f1 value: 0.25279084273189584 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (km) metrics: - type: accuracy value: 0.28295225285810355 - type: f1 value: 0.2664582563877155 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (kn) metrics: - type: accuracy value: 0.23480161398789506 - type: f1 value: 0.22275241866506734 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ko) metrics: - type: accuracy value: 0.3655682582380632 - type: f1 value: 0.3600475317106361 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (lv) metrics: - type: accuracy value: 0.4184936112979153 - type: f1 value: 0.4138932672359119 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ml) metrics: - type: accuracy value: 0.2490921318090114 - type: f1 value: 0.23968687483768808 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (mn) metrics: - type: accuracy value: 0.2986213853396099 - type: f1 value: 0.2997715207525541 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ms) metrics: - type: accuracy value: 0.4242098184263618 - type: f1 value: 0.4150877432664628 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (my) metrics: - type: accuracy value: 0.25131136516476127 - type: f1 value: 0.23938932214086775 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nb) metrics: - type: accuracy value: 0.3981506388702084 - type: f1 value: 0.3880958658779166 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nl) metrics: - type: accuracy value: 0.4362138533960995 - type: f1 value: 0.4201386842914633 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pl) metrics: - type: accuracy value: 0.4219569603227976 - type: f1 value: 0.4000556559825827 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pt) metrics: - type: accuracy value: 0.4520847343644923 - type: f1 value: 0.44241150050290506 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ro) metrics: - type: accuracy value: 0.4180901143241426 - type: f1 value: 0.40474074848670083 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ru) metrics: - type: accuracy value: 0.3596839273705447 - type: f1 value: 0.35095456843621 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sl) metrics: - type: accuracy value: 0.40605245460659045 - type: f1 value: 0.39302383051500134 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sq) metrics: - type: accuracy value: 0.42757229320780094 - type: f1 value: 0.41537639314973884 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sv) metrics: - type: accuracy value: 0.42347007397444514 - type: f1 value: 0.41043660179486263 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sw) metrics: - type: accuracy value: 0.4112306657700067 - type: f1 value: 0.39712940473289027 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ta) metrics: - type: accuracy value: 0.24603227975790182 - type: f1 value: 0.23969236788828607 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (te) metrics: - type: accuracy value: 0.2503698722259583 - type: f1 value: 0.2437196123281459 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (th) metrics: - type: accuracy value: 0.35400134498991254 - type: f1 value: 0.35063600413688034 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tl) metrics: - type: accuracy value: 0.4119031607262945 - type: f1 value: 0.4024043230427301 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tr) metrics: - type: accuracy value: 0.3640551445864156 - type: f1 value: 0.3603844992856558 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ur) metrics: - type: accuracy value: 0.25934767989240076 - type: f1 value: 0.252074457023531 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (vi) metrics: - type: accuracy value: 0.38799596503026224 - type: f1 value: 0.37160233794673125 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-CN) metrics: - type: accuracy value: 0.4624411566913248 - type: f1 value: 0.44367480561291905 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-TW) metrics: - type: accuracy value: 0.4230665770006724 - type: f1 value: 0.41964222328351397 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (af) metrics: - type: accuracy value: 0.43248150638870203 - type: f1 value: 0.40924230769590786 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (am) metrics: - type: accuracy value: 0.2530262273032952 - type: f1 value: 0.24937105830264067 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ar) metrics: - type: accuracy value: 0.3207128446536651 - type: f1 value: 0.3180245816594883 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (az) metrics: - type: accuracy value: 0.3668123739071957 - type: f1 value: 0.3637219042508338 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (bn) metrics: - type: accuracy value: 0.2956624075319435 - type: f1 value: 0.2838604205636276 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (cy) metrics: - type: accuracy value: 0.421049092131809 - type: f1 value: 0.38926150886991295 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (da) metrics: - type: accuracy value: 0.4544384667114997 - type: f1 value: 0.42578252395460003 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (de) metrics: - type: accuracy value: 0.43211163416274373 - type: f1 value: 0.41044658583047894 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (el) metrics: - type: accuracy value: 0.3650302622730329 - type: f1 value: 0.3449785095312759 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (en) metrics: - type: accuracy value: 0.6973772696704774 - type: f1 value: 0.6921759502909044 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (es) metrics: - type: accuracy value: 0.44078681909885675 - type: f1 value: 0.4305914426901129 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fa) metrics: - type: accuracy value: 0.32612642905178213 - type: f1 value: 0.3202463177462754 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fi) metrics: - type: accuracy value: 0.40356422326832553 - type: f1 value: 0.3813642481807678 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fr) metrics: - type: accuracy value: 0.4506724949562878 - type: f1 value: 0.4319827608343738 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (he) metrics: - type: accuracy value: 0.3217888365837256 - type: f1 value: 0.29979761884698775 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hi) metrics: - type: accuracy value: 0.26903160726294556 - type: f1 value: 0.25833010434083364 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hu) metrics: - type: accuracy value: 0.4037995965030262 - type: f1 value: 0.37931343552928826 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hy) metrics: - type: accuracy value: 0.28375924680564896 - type: f1 value: 0.2696255693013172 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (id) metrics: - type: accuracy value: 0.44361129791526566 - type: f1 value: 0.4354445012295126 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (is) metrics: - type: accuracy value: 0.39290517821116344 - type: f1 value: 0.3726982052174147 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (it) metrics: - type: accuracy value: 0.46469401479488903 - type: f1 value: 0.44060986162841564 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ja) metrics: - type: accuracy value: 0.46257565568258235 - type: f1 value: 0.4562513945675882 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (jv) metrics: - type: accuracy value: 0.41126429051782115 - type: f1 value: 0.3954392378396527 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ka) metrics: - type: accuracy value: 0.24727639542703428 - type: f1 value: 0.23337743140804484 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (km) metrics: - type: accuracy value: 0.2974108944182918 - type: f1 value: 0.2757087619008375 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (kn) metrics: - type: accuracy value: 0.23850033624747816 - type: f1 value: 0.2286733484540032 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ko) metrics: - type: accuracy value: 0.3656691324815064 - type: f1 value: 0.35504081677134564 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (lv) metrics: - type: accuracy value: 0.40928043039677203 - type: f1 value: 0.3910858913121125 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ml) metrics: - type: accuracy value: 0.25527908540685945 - type: f1 value: 0.25333391622280477 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (mn) metrics: - type: accuracy value: 0.29105581708137185 - type: f1 value: 0.28478235012692815 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ms) metrics: - type: accuracy value: 0.43786146603900467 - type: f1 value: 0.41964014392626703 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (my) metrics: - type: accuracy value: 0.27269670477471414 - type: f1 value: 0.26228386764141853 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nb) metrics: - type: accuracy value: 0.3901815736381977 - type: f1 value: 0.37641949339321856 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nl) metrics: - type: accuracy value: 0.4535978480161399 - type: f1 value: 0.426851176096831 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pl) metrics: - type: accuracy value: 0.41893073301950234 - type: f1 value: 0.4088871064261502 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pt) metrics: - type: accuracy value: 0.45901143241425685 - type: f1 value: 0.44496942353920543 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ro) metrics: - type: accuracy value: 0.44115669132481505 - type: f1 value: 0.41953945105870616 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ru) metrics: - type: accuracy value: 0.3276395427034297 - type: f1 value: 0.31436372571600935 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sl) metrics: - type: accuracy value: 0.40504371217215873 - type: f1 value: 0.39322752749628165 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sq) metrics: - type: accuracy value: 0.4251849361129792 - type: f1 value: 0.41413929711846303 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sv) metrics: - type: accuracy value: 0.42293207800941496 - type: f1 value: 0.4050409536806683 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sw) metrics: - type: accuracy value: 0.42999327505043705 - type: f1 value: 0.4104541622497327 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ta) metrics: - type: accuracy value: 0.2832548755884331 - type: f1 value: 0.2727684199556187 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (te) metrics: - type: accuracy value: 0.26593813046402154 - type: f1 value: 0.25483878616197586 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (th) metrics: - type: accuracy value: 0.36788836583725626 - type: f1 value: 0.34603932909177687 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tl) metrics: - type: accuracy value: 0.425689307330195 - type: f1 value: 0.40924469309079825 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tr) metrics: - type: accuracy value: 0.37094821788836585 - type: f1 value: 0.3794962882285716 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ur) metrics: - type: accuracy value: 0.2883658372562206 - type: f1 value: 0.2780655865551234 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (vi) metrics: - type: accuracy value: 0.37357094821788833 - type: f1 value: 0.3750791896103816 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-CN) metrics: - type: accuracy value: 0.4937794216543375 - type: f1 value: 0.4720421153697707 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-TW) metrics: - type: accuracy value: 0.44421654337592476 - type: f1 value: 0.4434741861198931 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-p2p name: MTEB MedrxivClusteringP2P metrics: - type: v_measure value: 0.31374938993074253 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-s2s name: MTEB MedrxivClusteringS2S metrics: - type: v_measure value: 0.26871455379644094 - task: type: Reranking dataset: type: mteb/mind_small name: MTEB MindSmallReranking metrics: - type: map value: 0.30402396942935334 - type: mrr value: 0.3142600938803256 - task: type: Retrieval dataset: type: nfcorpus name: MTEB NFCorpus metrics: - type: map_at_1 value: 0.03774 - type: map_at_10 value: 0.07615 - type: map_at_100 value: 0.09574 - type: map_at_1000 value: 0.10711 - type: map_at_3 value: 0.05754 - type: map_at_5 value: 0.06666 - type: mrr_at_1 value: 0.33127 - type: mrr_at_10 value: 0.40351 - type: mrr_at_100 value: 0.41144 - type: mrr_at_1000 value: 0.41202 - type: mrr_at_3 value: 0.38029 - type: mrr_at_5 value: 0.3919 - type: ndcg_at_1 value: 0.31579 - type: ndcg_at_10 value: 0.22792 - type: ndcg_at_100 value: 0.21699 - type: ndcg_at_1000 value: 0.30893 - type: ndcg_at_3 value: 0.26829 - type: ndcg_at_5 value: 0.25119 - type: precision_at_1 value: 0.33127 - type: precision_at_10 value: 0.16718 - type: precision_at_100 value: 0.05709 - type: precision_at_1000 value: 0.01836 - type: precision_at_3 value: 0.24768 - type: precision_at_5 value: 0.213 - type: recall_at_1 value: 0.03774 - type: recall_at_10 value: 0.10303 - type: recall_at_100 value: 0.23013 - type: recall_at_1000 value: 0.54865 - type: recall_at_3 value: 0.06554 - type: recall_at_5 value: 0.08087 - task: type: Retrieval dataset: type: nq name: MTEB NQ metrics: - type: map_at_1 value: 0.15621 - type: map_at_10 value: 0.24519 - type: map_at_100 value: 0.25586 - type: map_at_1000 value: 0.25662 - type: map_at_3 value: 0.21619 - type: map_at_5 value: 0.23232 - type: mrr_at_1 value: 0.17497 - type: mrr_at_10 value: 0.26301 - type: mrr_at_100 value: 0.27235 - type: mrr_at_1000 value: 0.27297 - type: mrr_at_3 value: 0.23561 - type: mrr_at_5 value: 0.25111 - type: ndcg_at_1 value: 0.17497 - type: ndcg_at_10 value: 0.29725 - type: ndcg_at_100 value: 0.34824 - type: ndcg_at_1000 value: 0.36907 - type: ndcg_at_3 value: 0.23946 - type: ndcg_at_5 value: 0.26739 - type: precision_at_1 value: 0.17497 - type: precision_at_10 value: 0.05217 - type: precision_at_100 value: 0.0081 - type: precision_at_1000 value: 0.00101 - type: precision_at_3 value: 0.11114 - type: precision_at_5 value: 0.08285 - type: recall_at_1 value: 0.15621 - type: recall_at_10 value: 0.43999 - type: recall_at_100 value: 0.67183 - type: recall_at_1000 value: 0.83174 - type: recall_at_3 value: 0.2872 - type: recall_at_5 value: 0.35154 - task: type: Retrieval dataset: type: quora name: MTEB QuoraRetrieval metrics: - type: map_at_1 value: 0.54717 - type: map_at_10 value: 0.67514 - type: map_at_100 value: 0.68484 - type: map_at_1000 value: 0.68523 - type: map_at_3 value: 0.64169 - type: map_at_5 value: 0.66054 - type: mrr_at_1 value: 0.6246 - type: mrr_at_10 value: 0.71503 - type: mrr_at_100 value: 0.71915 - type: mrr_at_1000 value: 0.71923 - type: mrr_at_3 value: 0.69468 - type: mrr_at_5 value: 0.70677 - type: ndcg_at_1 value: 0.6248 - type: ndcg_at_10 value: 0.7298 - type: ndcg_at_100 value: 0.76023 - type: ndcg_at_1000 value: 0.76512 - type: ndcg_at_3 value: 0.68138 - type: ndcg_at_5 value: 0.70458 - type: precision_at_1 value: 0.6248 - type: precision_at_10 value: 0.11373 - type: precision_at_100 value: 0.01437 - type: precision_at_1000 value: 0.00154 - type: precision_at_3 value: 0.29623 - type: precision_at_5 value: 0.19918 - type: recall_at_1 value: 0.54717 - type: recall_at_10 value: 0.84745 - type: recall_at_100 value: 0.96528 - type: recall_at_1000 value: 0.9939 - type: recall_at_3 value: 0.71606 - type: recall_at_5 value: 0.77511 - task: type: Clustering dataset: type: mteb/reddit-clustering name: MTEB RedditClustering metrics: - type: v_measure value: 0.4023390747226228 - task: type: Clustering dataset: type: mteb/reddit-clustering-p2p name: MTEB RedditClusteringP2P metrics: - type: v_measure value: 0.49090518272935624 - task: type: Retrieval dataset: type: scidocs name: MTEB SCIDOCS metrics: - type: map_at_1 value: 0.03028 - type: map_at_10 value: 0.06968 - type: map_at_100 value: 0.082 - type: map_at_1000 value: 0.08432 - type: map_at_3 value: 0.05307 - type: map_at_5 value: 0.06099 - type: mrr_at_1 value: 0.148 - type: mrr_at_10 value: 0.22425 - type: mrr_at_100 value: 0.23577 - type: mrr_at_1000 value: 0.2367 - type: mrr_at_3 value: 0.20233 - type: mrr_at_5 value: 0.21318 - type: ndcg_at_1 value: 0.148 - type: ndcg_at_10 value: 0.12206 - type: ndcg_at_100 value: 0.17799 - type: ndcg_at_1000 value: 0.22891 - type: ndcg_at_3 value: 0.12128 - type: ndcg_at_5 value: 0.10212 - type: precision_at_1 value: 0.148 - type: precision_at_10 value: 0.0617 - type: precision_at_100 value: 0.01428 - type: precision_at_1000 value: 0.00266 - type: precision_at_3 value: 0.11333 - type: precision_at_5 value: 0.0874 - type: recall_at_1 value: 0.03028 - type: recall_at_10 value: 0.12522 - type: recall_at_100 value: 0.28975 - type: recall_at_1000 value: 0.54038 - type: recall_at_3 value: 0.06913 - type: recall_at_5 value: 0.08883 - task: type: STS dataset: type: mteb/sickr-sts name: MTEB SICK-R metrics: - type: cos_sim_pearson value: 0.7662983928119752 - type: cos_sim_spearman value: 0.6592910683118656 - type: euclidean_pearson value: 0.7110290039690963 - type: euclidean_spearman value: 0.6480076622426653 - type: manhattan_pearson value: 0.708944726230188 - type: manhattan_spearman value: 0.6475082576033987 - task: type: STS dataset: type: mteb/sts12-sts name: MTEB STS12 metrics: - type: cos_sim_pearson value: 0.7442679147085554 - type: cos_sim_spearman value: 0.6652980061546658 - type: euclidean_pearson value: 0.7487039477408763 - type: euclidean_spearman value: 0.7063397666902785 - type: manhattan_pearson value: 0.7497015137513088 - type: manhattan_spearman value: 0.7075951355434326 - task: type: STS dataset: type: mteb/sts13-sts name: MTEB STS13 metrics: - type: cos_sim_pearson value: 0.7562472426599542 - type: cos_sim_spearman value: 0.7616628863742361 - type: euclidean_pearson value: 0.763297128081315 - type: euclidean_spearman value: 0.7719385151966562 - type: manhattan_pearson value: 0.7650363291423257 - type: manhattan_spearman value: 0.7737081896355399 - task: type: STS dataset: type: mteb/sts14-sts name: MTEB STS14 metrics: - type: cos_sim_pearson value: 0.7448227705407036 - type: cos_sim_spearman value: 0.6904572664009687 - type: euclidean_pearson value: 0.7176138185714849 - type: euclidean_spearman value: 0.6893415452043307 - type: manhattan_pearson value: 0.7168010915543306 - type: manhattan_spearman value: 0.6899176321262805 - task: type: STS dataset: type: mteb/sts15-sts name: MTEB STS15 metrics: - type: cos_sim_pearson value: 0.781566527175902 - type: cos_sim_spearman value: 0.7923677712825851 - type: euclidean_pearson value: 0.7629138438696417 - type: euclidean_spearman value: 0.7720108266215374 - type: manhattan_pearson value: 0.7627464935799118 - type: manhattan_spearman value: 0.7715286174478099 - task: type: STS dataset: type: mteb/sts16-sts name: MTEB STS16 metrics: - type: cos_sim_pearson value: 0.75068454465977 - type: cos_sim_spearman value: 0.7606792422441928 - type: euclidean_pearson value: 0.7064605440627698 - type: euclidean_spearman value: 0.7021776051117844 - type: manhattan_pearson value: 0.7032479295054919 - type: manhattan_spearman value: 0.6989782458638528 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (ko-ko) metrics: - type: cos_sim_pearson value: 0.39433272899394367 - type: cos_sim_spearman value: 0.5238601027550566 - type: euclidean_pearson value: 0.46409999048857453 - type: euclidean_spearman value: 0.5100333465175934 - type: manhattan_pearson value: 0.4655753533133655 - type: manhattan_spearman value: 0.5107550440519388 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (ar-ar) metrics: - type: cos_sim_pearson value: 0.5554431928210687 - type: cos_sim_spearman value: 0.5561674586076298 - type: euclidean_pearson value: 0.5807442713714088 - type: euclidean_spearman value: 0.5574066216931719 - type: manhattan_pearson value: 0.5784021675638542 - type: manhattan_spearman value: 0.5520365812536853 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-ar) metrics: - type: cos_sim_pearson value: 0.11378463868809098 - type: cos_sim_spearman value: 0.08209569244801064 - type: euclidean_pearson value: 0.0107041700730406 - type: euclidean_spearman value: -0.02205219710893189 - type: manhattan_pearson value: -0.007671300251104268 - type: manhattan_spearman value: -0.03430645020535567 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-de) metrics: - type: cos_sim_pearson value: 0.32714035609290126 - type: cos_sim_spearman value: 0.3018181775929109 - type: euclidean_pearson value: 0.2557368595910298 - type: euclidean_spearman value: 0.23316649115731378 - type: manhattan_pearson value: 0.24144200325329615 - type: manhattan_spearman value: 0.21646215463384572 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-en) metrics: - type: cos_sim_pearson value: 0.8336340470799157 - type: cos_sim_spearman value: 0.8495398260629698 - type: euclidean_pearson value: 0.8069876969911645 - type: euclidean_spearman value: 0.8097451731130427 - type: manhattan_pearson value: 0.8065869354146945 - type: manhattan_spearman value: 0.808540858718528 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-tr) metrics: - type: cos_sim_pearson value: 0.019200044163754912 - type: cos_sim_spearman value: 0.010393399782021343 - type: euclidean_pearson value: -0.011376003191297994 - type: euclidean_spearman value: -0.018947106671763914 - type: manhattan_pearson value: -0.038362564474484336 - type: manhattan_spearman value: -0.04242750882792888 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (es-en) metrics: - type: cos_sim_pearson value: 0.26561262451099577 - type: cos_sim_spearman value: 0.28776666666659906 - type: euclidean_pearson value: 0.14640410196999087 - type: euclidean_spearman value: 0.1610557011701786 - type: manhattan_pearson value: 0.15019405495911273 - type: manhattan_spearman value: 0.1537192083104197 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (es-es) metrics: - type: cos_sim_pearson value: 0.6975442020014331 - type: cos_sim_spearman value: 0.7188444295144646 - type: euclidean_pearson value: 0.7384934185952773 - type: euclidean_spearman value: 0.7326911108021089 - type: manhattan_pearson value: 0.7404354196954575 - type: manhattan_spearman value: 0.7337650787943871 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (fr-en) metrics: - type: cos_sim_pearson value: 0.2770511842301491 - type: cos_sim_spearman value: 0.26339466714066445 - type: euclidean_pearson value: 0.09323158236506385 - type: euclidean_spearman value: 0.0732083231520273 - type: manhattan_pearson value: 0.07807399527573071 - type: manhattan_spearman value: 0.05525546663067112 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (it-en) metrics: - type: cos_sim_pearson value: 0.24226521799447692 - type: cos_sim_spearman value: 0.20729929404589678 - type: euclidean_pearson value: 0.06753378617205011 - type: euclidean_spearman value: 0.06281654679029505 - type: manhattan_pearson value: 0.07087180250449322 - type: manhattan_spearman value: 0.0641611659259516 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (nl-en) metrics: - type: cos_sim_pearson value: 0.2913141236406123 - type: cos_sim_spearman value: 0.2505342961279355 - type: euclidean_pearson value: 0.10657141303961999 - type: euclidean_spearman value: 0.09712124819778453 - type: manhattan_pearson value: 0.12481782693315688 - type: manhattan_spearman value: 0.11287958480905973 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (en) metrics: - type: cos_sim_pearson value: 0.6404750650962878 - type: cos_sim_spearman value: 0.6566183708171826 - type: euclidean_pearson value: 0.6690887604405887 - type: euclidean_spearman value: 0.6689814072484552 - type: manhattan_pearson value: 0.6731627110509089 - type: manhattan_spearman value: 0.6701048176165322 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de) metrics: - type: cos_sim_pearson value: 0.1926519187000913 - type: cos_sim_spearman value: 0.21987647321429005 - type: euclidean_pearson value: 0.17850618752342948 - type: euclidean_spearman value: 0.2286669392885474 - type: manhattan_pearson value: 0.1816183594260708 - type: manhattan_spearman value: 0.23637510352837907 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es) metrics: - type: cos_sim_pearson value: 0.34221261828226934 - type: cos_sim_spearman value: 0.49811823238907665 - type: euclidean_pearson value: 0.4450394399762147 - type: euclidean_spearman value: 0.5095918449507287 - type: manhattan_pearson value: 0.4583191034038624 - type: manhattan_spearman value: 0.5019040986611795 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl) metrics: - type: cos_sim_pearson value: 0.03620381732096531 - type: cos_sim_spearman value: 0.23308439517991938 - type: euclidean_pearson value: 0.00965453312113125 - type: euclidean_spearman value: 0.24235967620790316 - type: manhattan_pearson value: 0.014408922275701605 - type: manhattan_spearman value: 0.25161920137046095 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (tr) metrics: - type: cos_sim_pearson value: 0.16694896287262667 - type: cos_sim_spearman value: 0.3466348380997687 - type: euclidean_pearson value: 0.29415825529188605 - type: euclidean_spearman value: 0.3833011033170646 - type: manhattan_pearson value: 0.3123273195263394 - type: manhattan_spearman value: 0.39100557857557955 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (ar) metrics: - type: cos_sim_pearson value: 0.09134927430889528 - type: cos_sim_spearman value: 0.2818922448944151 - type: euclidean_pearson value: 0.1986814169549051 - type: euclidean_spearman value: 0.2751958864494863 - type: manhattan_pearson value: 0.2180949221238945 - type: manhattan_spearman value: 0.2825217200494078 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (ru) metrics: - type: cos_sim_pearson value: 0.03638648294235208 - type: cos_sim_spearman value: 0.09068119621940965 - type: euclidean_pearson value: 0.008123129118737714 - type: euclidean_spearman value: 0.09173672890166147 - type: manhattan_pearson value: 0.007545188998226579 - type: manhattan_spearman value: 0.08431719541986525 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh) metrics: - type: cos_sim_pearson value: 0.029720915749084316 - type: cos_sim_spearman value: 0.2548511383289232 - type: euclidean_pearson value: 0.12751569670148918 - type: euclidean_spearman value: 0.24940721642439287 - type: manhattan_pearson value: 0.14310238482989826 - type: manhattan_spearman value: 0.2469821216148647 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr) metrics: - type: cos_sim_pearson value: 0.544745185734135 - type: cos_sim_spearman value: 0.6766493409568727 - type: euclidean_pearson value: 0.6013580336797049 - type: euclidean_spearman value: 0.6612319300814538 - type: manhattan_pearson value: 0.6081621036870816 - type: manhattan_spearman value: 0.6570010026716765 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-en) metrics: - type: cos_sim_pearson value: 0.49378654125882004 - type: cos_sim_spearman value: 0.5307135629778896 - type: euclidean_pearson value: 0.49292014167110915 - type: euclidean_spearman value: 0.5054523702399645 - type: manhattan_pearson value: 0.5126576414126853 - type: manhattan_spearman value: 0.5197908640319361 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es-en) metrics: - type: cos_sim_pearson value: 0.4492565239256213 - type: cos_sim_spearman value: 0.49512539047677256 - type: euclidean_pearson value: 0.4879346518897415 - type: euclidean_spearman value: 0.5147957870101565 - type: manhattan_pearson value: 0.49513145538980435 - type: manhattan_spearman value: 0.5189520789318917 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (it) metrics: - type: cos_sim_pearson value: 0.4524169032111187 - type: cos_sim_spearman value: 0.4824795739512037 - type: euclidean_pearson value: 0.49227194943998964 - type: euclidean_spearman value: 0.49641024420428087 - type: manhattan_pearson value: 0.4949788773297025 - type: manhattan_spearman value: 0.49940515338096303 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl-en) metrics: - type: cos_sim_pearson value: 0.3642138324083909 - type: cos_sim_spearman value: 0.36798674894178013 - type: euclidean_pearson value: 0.27760612942610086 - type: euclidean_spearman value: 0.29140966500287624 - type: manhattan_pearson value: 0.28456674031350115 - type: manhattan_spearman value: 0.2746356370924497 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh-en) metrics: - type: cos_sim_pearson value: 0.2655350664089358 - type: cos_sim_spearman value: 0.2868170719697501 - type: euclidean_pearson value: 0.12613577889195138 - type: euclidean_spearman value: 0.13589493311702933 - type: manhattan_pearson value: 0.11640157427420958 - type: manhattan_spearman value: 0.10345223941212416 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es-it) metrics: - type: cos_sim_pearson value: 0.3854682179114309 - type: cos_sim_spearman value: 0.45782560880405704 - type: euclidean_pearson value: 0.46496857002368486 - type: euclidean_spearman value: 0.4821270426410012 - type: manhattan_pearson value: 0.4687183911937405 - type: manhattan_spearman value: 0.4755698777385152 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-fr) metrics: - type: cos_sim_pearson value: 0.3512956772546032 - type: cos_sim_spearman value: 0.3296920218281008 - type: euclidean_pearson value: 0.3423140384382136 - type: euclidean_spearman value: 0.3219303153191447 - type: manhattan_pearson value: 0.3418946827660064 - type: manhattan_spearman value: 0.3488706570973238 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-pl) metrics: - type: cos_sim_pearson value: 0.30507667380509634 - type: cos_sim_spearman value: 0.20447284723752715 - type: euclidean_pearson value: 0.29662041381794474 - type: euclidean_spearman value: 0.20939990379746756 - type: manhattan_pearson value: 0.32511208050632806 - type: manhattan_spearman value: 0.23773047901712493 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr-pl) metrics: - type: cos_sim_pearson value: 0.7110820459712155 - type: cos_sim_spearman value: 0.6197797868009122 - type: euclidean_pearson value: 0.6030910689156633 - type: euclidean_spearman value: 0.6197797868009122 - type: manhattan_pearson value: 0.6634051769640379 - type: manhattan_spearman value: 0.6197797868009122 - task: type: STS dataset: type: mteb/stsbenchmark-sts name: MTEB STSBenchmark metrics: - type: cos_sim_pearson value: 0.7653032504460737 - type: cos_sim_spearman value: 0.7533716094627373 - type: euclidean_pearson value: 0.6964662673290599 - type: euclidean_spearman value: 0.6730188896368857 - type: manhattan_pearson value: 0.6945096082050808 - type: manhattan_spearman value: 0.670718727259371 - task: type: Reranking dataset: type: mteb/scidocs-reranking name: MTEB SciDocsRR metrics: - type: map value: 0.7133941904192648 - type: mrr value: 0.8973766429648782 - task: type: Retrieval dataset: type: scifact name: MTEB SciFact metrics: - type: map_at_1 value: 0.43333 - type: map_at_10 value: 0.52364 - type: map_at_100 value: 0.53184 - type: map_at_1000 value: 0.53234 - type: map_at_3 value: 0.49832 - type: map_at_5 value: 0.51244 - type: mrr_at_1 value: 0.45333 - type: mrr_at_10 value: 0.53455 - type: mrr_at_100 value: 0.54191 - type: mrr_at_1000 value: 0.54235 - type: mrr_at_3 value: 0.51556 - type: mrr_at_5 value: 0.52622 - type: ndcg_at_1 value: 0.45333 - type: ndcg_at_10 value: 0.56899 - type: ndcg_at_100 value: 0.60702 - type: ndcg_at_1000 value: 0.62046 - type: ndcg_at_3 value: 0.52451 - type: ndcg_at_5 value: 0.54534 - type: precision_at_1 value: 0.45333 - type: precision_at_10 value: 0.078 - type: precision_at_100 value: 0.00987 - type: precision_at_1000 value: 0.0011 - type: precision_at_3 value: 0.20778 - type: precision_at_5 value: 0.13867 - type: recall_at_1 value: 0.43333 - type: recall_at_10 value: 0.697 - type: recall_at_100 value: 0.869 - type: recall_at_1000 value: 0.976 - type: recall_at_3 value: 0.57817 - type: recall_at_5 value: 0.62828 - task: type: PairClassification dataset: type: mteb/sprintduplicatequestions-pairclassification name: MTEB SprintDuplicateQuestions metrics: - type: cos_sim_accuracy value: 0.997 - type: cos_sim_accuracy_threshold value: 0.7497431039810181 - type: cos_sim_ap value: 0.8988577913120002 - type: cos_sim_f1 value: 0.8462694041061593 - type: cos_sim_f1_threshold value: 0.7372856140136719 - type: cos_sim_precision value: 0.847542627883651 - type: cos_sim_recall value: 0.845 - type: dot_accuracy value: 0.9924752475247525 - type: dot_accuracy_threshold value: 1300.7469482421875 - type: dot_ap value: 0.5681855467290009 - type: dot_f1 value: 0.5608412618928393 - type: dot_f1_threshold value: 1153.91796875 - type: dot_precision value: 0.5616850551654965 - type: dot_recall value: 0.56 - type: euclidean_accuracy value: 0.9970594059405941 - type: euclidean_accuracy_threshold value: 26.410518646240234 - type: euclidean_ap value: 0.9012451226491524 - type: euclidean_f1 value: 0.8444211629125197 - type: euclidean_f1_threshold value: 26.42844581604004 - type: euclidean_precision value: 0.8866886688668867 - type: euclidean_recall value: 0.806 - type: manhattan_accuracy value: 0.9971287128712871 - type: manhattan_accuracy_threshold value: 548.025634765625 - type: manhattan_ap value: 0.9067590584183216 - type: manhattan_f1 value: 0.8485436893203884 - type: manhattan_f1_threshold value: 597.0555419921875 - type: manhattan_precision value: 0.8245283018867925 - type: manhattan_recall value: 0.874 - type: max_accuracy value: 0.9971287128712871 - type: max_ap value: 0.9067590584183216 - type: max_f1 value: 0.8485436893203884 - task: type: Clustering dataset: type: mteb/stackexchange-clustering name: MTEB StackExchangeClustering metrics: - type: v_measure value: 0.5274481093815175 - task: type: Clustering dataset: type: mteb/stackexchange-clustering-p2p name: MTEB StackExchangeClusteringP2P metrics: - type: v_measure value: 0.3265999453562101 - task: type: Reranking dataset: type: mteb/stackoverflowdupquestions-reranking name: MTEB StackOverflowDupQuestions metrics: - type: map value: 0.44744984645554653 - type: mrr value: 0.45333879764026824 - task: type: Summarization dataset: type: mteb/summeval name: MTEB SummEval metrics: - type: cos_sim_pearson value: 0.295961822471627 - type: cos_sim_spearman value: 0.2890145030911965 - type: dot_pearson value: 0.2917474339962901 - type: dot_spearman value: 0.2736297597081396 - task: type: Retrieval dataset: type: trec-covid name: MTEB TRECCOVID metrics: - type: map_at_1 value: 0.00241 - type: map_at_10 value: 0.01672 - type: map_at_100 value: 0.07859 - type: map_at_1000 value: 0.17616 - type: map_at_3 value: 0.00631 - type: map_at_5 value: 0.00968 - type: mrr_at_1 value: 0.9 - type: mrr_at_10 value: 0.92952 - type: mrr_at_100 value: 0.93036 - type: mrr_at_1000 value: 0.93036 - type: mrr_at_3 value: 0.92667 - type: mrr_at_5 value: 0.92667 - type: ndcg_at_1 value: 0.83 - type: ndcg_at_10 value: 0.70302 - type: ndcg_at_100 value: 0.48149 - type: ndcg_at_1000 value: 0.40709 - type: ndcg_at_3 value: 0.79173 - type: ndcg_at_5 value: 0.75347 - type: precision_at_1 value: 0.9 - type: precision_at_10 value: 0.726 - type: precision_at_100 value: 0.4846 - type: precision_at_1000 value: 0.18094 - type: precision_at_3 value: 0.84 - type: precision_at_5 value: 0.788 - type: recall_at_1 value: 0.00241 - type: recall_at_10 value: 0.01814 - type: recall_at_100 value: 0.11141 - type: recall_at_1000 value: 0.37709 - type: recall_at_3 value: 0.00647 - type: recall_at_5 value: 0.01015 - task: type: Retrieval dataset: type: webis-touche2020 name: MTEB Touche2020 metrics: - type: map_at_1 value: 0.02782 - type: map_at_10 value: 0.0906 - type: map_at_100 value: 0.14571 - type: map_at_1000 value: 0.16007 - type: map_at_3 value: 0.05037 - type: map_at_5 value: 0.0663 - type: mrr_at_1 value: 0.34694 - type: mrr_at_10 value: 0.48243 - type: mrr_at_100 value: 0.49065 - type: mrr_at_1000 value: 0.49065 - type: mrr_at_3 value: 0.44898 - type: mrr_at_5 value: 0.46429 - type: ndcg_at_1 value: 0.31633 - type: ndcg_at_10 value: 0.22972 - type: ndcg_at_100 value: 0.34777 - type: ndcg_at_1000 value: 0.45639 - type: ndcg_at_3 value: 0.26398 - type: ndcg_at_5 value: 0.24418 - type: precision_at_1 value: 0.34694 - type: precision_at_10 value: 0.19796 - type: precision_at_100 value: 0.07224 - type: precision_at_1000 value: 0.01445 - type: precision_at_3 value: 0.26531 - type: precision_at_5 value: 0.23265 - type: recall_at_1 value: 0.02782 - type: recall_at_10 value: 0.14841 - type: recall_at_100 value: 0.4486 - type: recall_at_1000 value: 0.78227 - type: recall_at_3 value: 0.05959 - type: recall_at_5 value: 0.08969 - task: type: Classification dataset: type: mteb/toxic_conversations_50k name: MTEB ToxicConversationsClassification metrics: - type: accuracy value: 0.6265799999999999 - type: ap value: 0.1096353161716344 - type: f1 value: 0.48294226423442643 - task: type: Classification dataset: type: mteb/tweet_sentiment_extraction name: MTEB TweetSentimentExtractionClassification metrics: - type: accuracy value: 0.5240803621958121 - type: f1 value: 0.5261009636022186 - task: type: Clustering dataset: type: mteb/twentynewsgroups-clustering name: MTEB TwentyNewsgroupsClustering metrics: - type: v_measure value: 0.3212697126747911 - task: type: PairClassification dataset: type: mteb/twittersemeval2015-pairclassification name: MTEB TwitterSemEval2015 metrics: - type: cos_sim_accuracy value: 0.8069976753889253 - type: cos_sim_accuracy_threshold value: 0.760696530342102 - type: cos_sim_ap value: 0.5474680676121269 - type: cos_sim_f1 value: 0.5318923998590391 - type: cos_sim_f1_threshold value: 0.6627046465873718 - type: cos_sim_precision value: 0.47935634130849036 - type: cos_sim_recall value: 0.5973614775725594 - type: dot_accuracy value: 0.7933480360016689 - type: dot_accuracy_threshold value: 1049.509033203125 - type: dot_ap value: 0.48469021289336267 - type: dot_f1 value: 0.5048010973936901 - type: dot_f1_threshold value: 762.2753295898438 - type: dot_precision value: 0.4206084051345173 - type: dot_recall value: 0.6311345646437995 - type: euclidean_accuracy value: 0.7978780473266973 - type: euclidean_accuracy_threshold value: 22.804443359375 - type: euclidean_ap value: 0.5025832725516481 - type: euclidean_f1 value: 0.49655838666827684 - type: euclidean_f1_threshold value: 28.18334197998047 - type: euclidean_precision value: 0.4578044978846582 - type: euclidean_recall value: 0.5424802110817942 - type: manhattan_accuracy value: 0.797699231090183 - type: manhattan_accuracy_threshold value: 473.0945129394531 - type: manhattan_ap value: 0.49898924857143634 - type: manhattan_f1 value: 0.4933043378734119 - type: manhattan_f1_threshold value: 604.353759765625 - type: manhattan_precision value: 0.43561754598746716 - type: manhattan_recall value: 0.5686015831134564 - type: max_accuracy value: 0.8069976753889253 - type: max_ap value: 0.5474680676121269 - type: max_f1 value: 0.5318923998590391 - task: type: PairClassification dataset: type: mteb/twitterurlcorpus-pairclassification name: MTEB TwitterURLCorpus metrics: - type: cos_sim_accuracy value: 0.8690573213800598 - type: cos_sim_accuracy_threshold value: 0.6966493129730225 - type: cos_sim_ap value: 0.8105760818661524 - type: cos_sim_f1 value: 0.7364688856729379 - type: cos_sim_f1_threshold value: 0.6471728086471558 - type: cos_sim_precision value: 0.6946491946491946 - type: cos_sim_recall value: 0.7836464428703419 - type: dot_accuracy value: 0.8380680715644041 - type: dot_accuracy_threshold value: 934.6768798828125 - type: dot_ap value: 0.724977400594746 - type: dot_f1 value: 0.6868460650173216 - type: dot_f1_threshold value: 830.3577270507812 - type: dot_precision value: 0.6295464750785811 - type: dot_recall value: 0.7556205728364644 - type: euclidean_accuracy value: 0.8597430822369697 - type: euclidean_accuracy_threshold value: 27.972591400146484 - type: euclidean_ap value: 0.7886101740829327 - type: euclidean_f1 value: 0.7107960824663695 - type: euclidean_f1_threshold value: 29.554906845092773 - type: euclidean_precision value: 0.7036897306270279 - type: euclidean_recall value: 0.718047428395442 - type: manhattan_accuracy value: 0.8594132029339854 - type: manhattan_accuracy_threshold value: 588.2883911132812 - type: manhattan_ap value: 0.7877876711171923 - type: manhattan_f1 value: 0.7107869075515911 - type: manhattan_f1_threshold value: 626.1203002929688 - type: manhattan_precision value: 0.6980697847067557 - type: manhattan_recall value: 0.7239759778256852 - type: max_accuracy value: 0.8690573213800598 - type: max_ap value: 0.8105760818661524 - type: max_f1 value: 0.7364688856729379 --- # SGPT-125M-weightedmean-msmarco-specb-bitfit ## Usage For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt ## Evaluation Results For eval results, refer to the eval folder or our paper: https://arxiv.org/abs/2202.08904 ## Training The model was trained with the parameters: **DataLoader**: `torch.utils.data.dataloader.DataLoader` of length 15600 with parameters: ``` {'batch_size': 32, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'} ``` **Loss**: `sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters: ``` {'scale': 20.0, 'similarity_fct': 'cos_sim'} ``` Parameters of the fit()-Method: ``` { "epochs": 10, "evaluation_steps": 0, "evaluator": "NoneType", "max_grad_norm": 1, "optimizer_class": "", "optimizer_params": { "lr": 0.0002 }, "scheduler": "WarmupLinear", "steps_per_epoch": null, "warmup_steps": 1000, "weight_decay": 0.01 } ``` ## Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: GPTNeoModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False}) ) ``` ## Citing & Authors ```bibtex @article{muennighoff2022sgpt, title={SGPT: GPT Sentence Embeddings for Semantic Search}, author={Muennighoff, Niklas}, journal={arXiv preprint arXiv:2202.08904}, year={2022} } ```