Muennighoff's picture
Update README.md (#1)
88aafd9
metadata
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - feature-extraction
  - sentence-similarity
  - mteb
model-index:
  - name: SGPT-5.8B-weightedmean-nli-bitfit
    results:
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_counterfactual
          name: MTEB AmazonCounterfactualClassification (en)
          config: en
          split: test
          revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
        metrics:
          - type: accuracy
            value: 74.07462686567165
          - type: ap
            value: 37.44692407529112
          - type: f1
            value: 68.28971003916419
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_counterfactual
          name: MTEB AmazonCounterfactualClassification (de)
          config: de
          split: test
          revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
        metrics:
          - type: accuracy
            value: 66.63811563169165
          - type: ap
            value: 78.57252079915924
          - type: f1
            value: 64.5543087846584
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_counterfactual
          name: MTEB AmazonCounterfactualClassification (en-ext)
          config: en-ext
          split: test
          revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
        metrics:
          - type: accuracy
            value: 77.21889055472263
          - type: ap
            value: 25.663426367826712
          - type: f1
            value: 64.26265688503176
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_counterfactual
          name: MTEB AmazonCounterfactualClassification (ja)
          config: ja
          split: test
          revision: 2d8a100785abf0ae21420d2a55b0c56e3e1ea996
        metrics:
          - type: accuracy
            value: 58.06209850107067
          - type: ap
            value: 14.028219107023915
          - type: f1
            value: 48.10387189660778
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_polarity
          name: MTEB AmazonPolarityClassification
          config: default
          split: test
          revision: 80714f8dcf8cefc218ef4f8c5a966dd83f75a0e1
        metrics:
          - type: accuracy
            value: 82.30920000000002
          - type: ap
            value: 76.88786578621213
          - type: f1
            value: 82.15455656065011
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_reviews_multi
          name: MTEB AmazonReviewsClassification (en)
          config: en
          split: test
          revision: c379a6705fec24a2493fa68e011692605f44e119
        metrics:
          - type: accuracy
            value: 41.584
          - type: f1
            value: 41.203137944390114
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_reviews_multi
          name: MTEB AmazonReviewsClassification (de)
          config: de
          split: test
          revision: c379a6705fec24a2493fa68e011692605f44e119
        metrics:
          - type: accuracy
            value: 35.288000000000004
          - type: f1
            value: 34.672995558518096
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_reviews_multi
          name: MTEB AmazonReviewsClassification (es)
          config: es
          split: test
          revision: c379a6705fec24a2493fa68e011692605f44e119
        metrics:
          - type: accuracy
            value: 38.34
          - type: f1
            value: 37.608755629529455
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_reviews_multi
          name: MTEB AmazonReviewsClassification (fr)
          config: fr
          split: test
          revision: c379a6705fec24a2493fa68e011692605f44e119
        metrics:
          - type: accuracy
            value: 37.839999999999996
          - type: f1
            value: 36.86898201563507
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_reviews_multi
          name: MTEB AmazonReviewsClassification (ja)
          config: ja
          split: test
          revision: c379a6705fec24a2493fa68e011692605f44e119
        metrics:
          - type: accuracy
            value: 30.936000000000003
          - type: f1
            value: 30.49401738527071
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_reviews_multi
          name: MTEB AmazonReviewsClassification (zh)
          config: zh
          split: test
          revision: c379a6705fec24a2493fa68e011692605f44e119
        metrics:
          - type: accuracy
            value: 33.75
          - type: f1
            value: 33.38338946025617
      - task:
          type: Retrieval
        dataset:
          type: arguana
          name: MTEB ArguAna
          config: default
          split: test
          revision: 5b3e3697907184a9b77a3c99ee9ea1a9cbb1e4e3
        metrics:
          - type: map_at_1
            value: 13.727
          - type: map_at_10
            value: 26.740000000000002
          - type: map_at_100
            value: 28.218
          - type: map_at_1000
            value: 28.246
          - type: map_at_3
            value: 21.728
          - type: map_at_5
            value: 24.371000000000002
          - type: ndcg_at_1
            value: 13.727
          - type: ndcg_at_10
            value: 35.07
          - type: ndcg_at_100
            value: 41.947
          - type: ndcg_at_1000
            value: 42.649
          - type: ndcg_at_3
            value: 24.484
          - type: ndcg_at_5
            value: 29.282999999999998
          - type: precision_at_1
            value: 13.727
          - type: precision_at_10
            value: 6.223
          - type: precision_at_100
            value: 0.9369999999999999
          - type: precision_at_1000
            value: 0.099
          - type: precision_at_3
            value: 10.835
          - type: precision_at_5
            value: 8.848
          - type: recall_at_1
            value: 13.727
          - type: recall_at_10
            value: 62.233000000000004
          - type: recall_at_100
            value: 93.67
          - type: recall_at_1000
            value: 99.14699999999999
          - type: recall_at_3
            value: 32.504
          - type: recall_at_5
            value: 44.239
      - task:
          type: Clustering
        dataset:
          type: mteb/arxiv-clustering-p2p
          name: MTEB ArxivClusteringP2P
          config: default
          split: test
          revision: 0bbdb47bcbe3a90093699aefeed338a0f28a7ee8
        metrics:
          - type: v_measure
            value: 40.553923271901695
      - task:
          type: Clustering
        dataset:
          type: mteb/arxiv-clustering-s2s
          name: MTEB ArxivClusteringS2S
          config: default
          split: test
          revision: b73bd54100e5abfa6e3a23dcafb46fe4d2438dc3
        metrics:
          - type: v_measure
            value: 32.49323183712211
      - task:
          type: Reranking
        dataset:
          type: mteb/askubuntudupquestions-reranking
          name: MTEB AskUbuntuDupQuestions
          config: default
          split: test
          revision: 4d853f94cd57d85ec13805aeeac3ae3e5eb4c49c
        metrics:
          - type: map
            value: 55.89811361443445
          - type: mrr
            value: 70.16235764850724
      - task:
          type: STS
        dataset:
          type: mteb/biosses-sts
          name: MTEB BIOSSES
          config: default
          split: test
          revision: 9ee918f184421b6bd48b78f6c714d86546106103
        metrics:
          - type: cos_sim_pearson
            value: 82.50506557805856
          - type: cos_sim_spearman
            value: 79.50000423261176
          - type: euclidean_pearson
            value: 75.76190885392926
          - type: euclidean_spearman
            value: 76.7330737163434
          - type: manhattan_pearson
            value: 75.825318036112
          - type: manhattan_spearman
            value: 76.7415076434559
      - task:
          type: BitextMining
        dataset:
          type: mteb/bucc-bitext-mining
          name: MTEB BUCC (de-en)
          config: de-en
          split: test
          revision: d51519689f32196a32af33b075a01d0e7c51e252
        metrics:
          - type: accuracy
            value: 75.49060542797494
          - type: f1
            value: 75.15379262352123
          - type: precision
            value: 74.99391092553932
          - type: recall
            value: 75.49060542797494
      - task:
          type: BitextMining
        dataset:
          type: mteb/bucc-bitext-mining
          name: MTEB BUCC (fr-en)
          config: fr-en
          split: test
          revision: d51519689f32196a32af33b075a01d0e7c51e252
        metrics:
          - type: accuracy
            value: 0.4182258419546555
          - type: f1
            value: 0.4182258419546555
          - type: precision
            value: 0.4182258419546555
          - type: recall
            value: 0.4182258419546555
      - task:
          type: BitextMining
        dataset:
          type: mteb/bucc-bitext-mining
          name: MTEB BUCC (ru-en)
          config: ru-en
          split: test
          revision: d51519689f32196a32af33b075a01d0e7c51e252
        metrics:
          - type: accuracy
            value: 0.013855213023900243
          - type: f1
            value: 0.0115460108532502
          - type: precision
            value: 0.010391409767925183
          - type: recall
            value: 0.013855213023900243
      - task:
          type: BitextMining
        dataset:
          type: mteb/bucc-bitext-mining
          name: MTEB BUCC (zh-en)
          config: zh-en
          split: test
          revision: d51519689f32196a32af33b075a01d0e7c51e252
        metrics:
          - type: accuracy
            value: 0.315955766192733
          - type: f1
            value: 0.315955766192733
          - type: precision
            value: 0.315955766192733
          - type: recall
            value: 0.315955766192733
      - task:
          type: Classification
        dataset:
          type: mteb/banking77
          name: MTEB Banking77Classification
          config: default
          split: test
          revision: 44fa15921b4c889113cc5df03dd4901b49161ab7
        metrics:
          - type: accuracy
            value: 81.74025974025973
          - type: f1
            value: 81.66568824876
      - task:
          type: Clustering
        dataset:
          type: mteb/biorxiv-clustering-p2p
          name: MTEB BiorxivClusteringP2P
          config: default
          split: test
          revision: 11d0121201d1f1f280e8cc8f3d98fb9c4d9f9c55
        metrics:
          - type: v_measure
            value: 33.59451202614059
      - task:
          type: Clustering
        dataset:
          type: mteb/biorxiv-clustering-s2s
          name: MTEB BiorxivClusteringS2S
          config: default
          split: test
          revision: c0fab014e1bcb8d3a5e31b2088972a1e01547dc1
        metrics:
          - type: v_measure
            value: 29.128241446157165
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackAndroidRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 26.715
          - type: map_at_10
            value: 35.007
          - type: map_at_100
            value: 36.352000000000004
          - type: map_at_1000
            value: 36.51
          - type: map_at_3
            value: 32.257999999999996
          - type: map_at_5
            value: 33.595000000000006
          - type: ndcg_at_1
            value: 33.906
          - type: ndcg_at_10
            value: 40.353
          - type: ndcg_at_100
            value: 45.562999999999995
          - type: ndcg_at_1000
            value: 48.454
          - type: ndcg_at_3
            value: 36.349
          - type: ndcg_at_5
            value: 37.856
          - type: precision_at_1
            value: 33.906
          - type: precision_at_10
            value: 7.854
          - type: precision_at_100
            value: 1.29
          - type: precision_at_1000
            value: 0.188
          - type: precision_at_3
            value: 17.549
          - type: precision_at_5
            value: 12.561
          - type: recall_at_1
            value: 26.715
          - type: recall_at_10
            value: 49.508
          - type: recall_at_100
            value: 71.76599999999999
          - type: recall_at_1000
            value: 91.118
          - type: recall_at_3
            value: 37.356
          - type: recall_at_5
            value: 41.836
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackEnglishRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 19.663
          - type: map_at_10
            value: 27.086
          - type: map_at_100
            value: 28.066999999999997
          - type: map_at_1000
            value: 28.18
          - type: map_at_3
            value: 24.819
          - type: map_at_5
            value: 26.332
          - type: ndcg_at_1
            value: 25.732
          - type: ndcg_at_10
            value: 31.613999999999997
          - type: ndcg_at_100
            value: 35.757
          - type: ndcg_at_1000
            value: 38.21
          - type: ndcg_at_3
            value: 28.332
          - type: ndcg_at_5
            value: 30.264000000000003
          - type: precision_at_1
            value: 25.732
          - type: precision_at_10
            value: 6.038
          - type: precision_at_100
            value: 1.034
          - type: precision_at_1000
            value: 0.149
          - type: precision_at_3
            value: 13.864
          - type: precision_at_5
            value: 10.241999999999999
          - type: recall_at_1
            value: 19.663
          - type: recall_at_10
            value: 39.585
          - type: recall_at_100
            value: 57.718
          - type: recall_at_1000
            value: 74.26700000000001
          - type: recall_at_3
            value: 29.845
          - type: recall_at_5
            value: 35.105
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackGamingRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 30.125
          - type: map_at_10
            value: 39.824
          - type: map_at_100
            value: 40.935
          - type: map_at_1000
            value: 41.019
          - type: map_at_3
            value: 37.144
          - type: map_at_5
            value: 38.647999999999996
          - type: ndcg_at_1
            value: 34.922
          - type: ndcg_at_10
            value: 45.072
          - type: ndcg_at_100
            value: 50.046
          - type: ndcg_at_1000
            value: 51.895
          - type: ndcg_at_3
            value: 40.251
          - type: ndcg_at_5
            value: 42.581
          - type: precision_at_1
            value: 34.922
          - type: precision_at_10
            value: 7.303999999999999
          - type: precision_at_100
            value: 1.0739999999999998
          - type: precision_at_1000
            value: 0.13
          - type: precision_at_3
            value: 17.994
          - type: precision_at_5
            value: 12.475999999999999
          - type: recall_at_1
            value: 30.125
          - type: recall_at_10
            value: 57.253
          - type: recall_at_100
            value: 79.35799999999999
          - type: recall_at_1000
            value: 92.523
          - type: recall_at_3
            value: 44.088
          - type: recall_at_5
            value: 49.893
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackGisRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 16.298000000000002
          - type: map_at_10
            value: 21.479
          - type: map_at_100
            value: 22.387
          - type: map_at_1000
            value: 22.483
          - type: map_at_3
            value: 19.743
          - type: map_at_5
            value: 20.444000000000003
          - type: ndcg_at_1
            value: 17.740000000000002
          - type: ndcg_at_10
            value: 24.887
          - type: ndcg_at_100
            value: 29.544999999999998
          - type: ndcg_at_1000
            value: 32.417
          - type: ndcg_at_3
            value: 21.274
          - type: ndcg_at_5
            value: 22.399
          - type: precision_at_1
            value: 17.740000000000002
          - type: precision_at_10
            value: 3.932
          - type: precision_at_100
            value: 0.666
          - type: precision_at_1000
            value: 0.094
          - type: precision_at_3
            value: 8.927
          - type: precision_at_5
            value: 6.056
          - type: recall_at_1
            value: 16.298000000000002
          - type: recall_at_10
            value: 34.031
          - type: recall_at_100
            value: 55.769000000000005
          - type: recall_at_1000
            value: 78.19500000000001
          - type: recall_at_3
            value: 23.799999999999997
          - type: recall_at_5
            value: 26.562
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackMathematicaRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 10.958
          - type: map_at_10
            value: 16.999
          - type: map_at_100
            value: 17.979
          - type: map_at_1000
            value: 18.112000000000002
          - type: map_at_3
            value: 15.010000000000002
          - type: map_at_5
            value: 16.256999999999998
          - type: ndcg_at_1
            value: 14.179
          - type: ndcg_at_10
            value: 20.985
          - type: ndcg_at_100
            value: 26.216
          - type: ndcg_at_1000
            value: 29.675
          - type: ndcg_at_3
            value: 17.28
          - type: ndcg_at_5
            value: 19.301
          - type: precision_at_1
            value: 14.179
          - type: precision_at_10
            value: 3.968
          - type: precision_at_100
            value: 0.784
          - type: precision_at_1000
            value: 0.121
          - type: precision_at_3
            value: 8.541
          - type: precision_at_5
            value: 6.468
          - type: recall_at_1
            value: 10.958
          - type: recall_at_10
            value: 29.903000000000002
          - type: recall_at_100
            value: 53.413
          - type: recall_at_1000
            value: 78.74799999999999
          - type: recall_at_3
            value: 19.717000000000002
          - type: recall_at_5
            value: 24.817
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackPhysicsRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 21.217
          - type: map_at_10
            value: 29.677
          - type: map_at_100
            value: 30.928
          - type: map_at_1000
            value: 31.063000000000002
          - type: map_at_3
            value: 26.611
          - type: map_at_5
            value: 28.463
          - type: ndcg_at_1
            value: 26.083000000000002
          - type: ndcg_at_10
            value: 35.217
          - type: ndcg_at_100
            value: 40.715
          - type: ndcg_at_1000
            value: 43.559
          - type: ndcg_at_3
            value: 30.080000000000002
          - type: ndcg_at_5
            value: 32.701
          - type: precision_at_1
            value: 26.083000000000002
          - type: precision_at_10
            value: 6.622
          - type: precision_at_100
            value: 1.115
          - type: precision_at_1000
            value: 0.156
          - type: precision_at_3
            value: 14.629
          - type: precision_at_5
            value: 10.837
          - type: recall_at_1
            value: 21.217
          - type: recall_at_10
            value: 47.031
          - type: recall_at_100
            value: 70.378
          - type: recall_at_1000
            value: 89.704
          - type: recall_at_3
            value: 32.427
          - type: recall_at_5
            value: 39.31
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackProgrammersRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 19.274
          - type: map_at_10
            value: 26.398
          - type: map_at_100
            value: 27.711000000000002
          - type: map_at_1000
            value: 27.833000000000002
          - type: map_at_3
            value: 24.294
          - type: map_at_5
            value: 25.385
          - type: ndcg_at_1
            value: 24.886
          - type: ndcg_at_10
            value: 30.909
          - type: ndcg_at_100
            value: 36.941
          - type: ndcg_at_1000
            value: 39.838
          - type: ndcg_at_3
            value: 27.455000000000002
          - type: ndcg_at_5
            value: 28.828
          - type: precision_at_1
            value: 24.886
          - type: precision_at_10
            value: 5.6739999999999995
          - type: precision_at_100
            value: 1.0290000000000001
          - type: precision_at_1000
            value: 0.146
          - type: precision_at_3
            value: 13.242
          - type: precision_at_5
            value: 9.292
          - type: recall_at_1
            value: 19.274
          - type: recall_at_10
            value: 39.643
          - type: recall_at_100
            value: 66.091
          - type: recall_at_1000
            value: 86.547
          - type: recall_at_3
            value: 29.602
          - type: recall_at_5
            value: 33.561
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 18.653666666666666
          - type: map_at_10
            value: 25.606666666666666
          - type: map_at_100
            value: 26.669333333333334
          - type: map_at_1000
            value: 26.795833333333334
          - type: map_at_3
            value: 23.43433333333333
          - type: map_at_5
            value: 24.609666666666666
          - type: ndcg_at_1
            value: 22.742083333333333
          - type: ndcg_at_10
            value: 29.978333333333335
          - type: ndcg_at_100
            value: 34.89808333333333
          - type: ndcg_at_1000
            value: 37.806583333333336
          - type: ndcg_at_3
            value: 26.223666666666674
          - type: ndcg_at_5
            value: 27.91033333333333
          - type: precision_at_1
            value: 22.742083333333333
          - type: precision_at_10
            value: 5.397083333333334
          - type: precision_at_100
            value: 0.9340000000000002
          - type: precision_at_1000
            value: 0.13691666666666663
          - type: precision_at_3
            value: 12.331083333333332
          - type: precision_at_5
            value: 8.805499999999999
          - type: recall_at_1
            value: 18.653666666666666
          - type: recall_at_10
            value: 39.22625000000001
          - type: recall_at_100
            value: 61.31049999999999
          - type: recall_at_1000
            value: 82.19058333333334
          - type: recall_at_3
            value: 28.517333333333333
          - type: recall_at_5
            value: 32.9565
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackStatsRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 16.07
          - type: map_at_10
            value: 21.509
          - type: map_at_100
            value: 22.335
          - type: map_at_1000
            value: 22.437
          - type: map_at_3
            value: 19.717000000000002
          - type: map_at_5
            value: 20.574
          - type: ndcg_at_1
            value: 18.865000000000002
          - type: ndcg_at_10
            value: 25.135999999999996
          - type: ndcg_at_100
            value: 29.483999999999998
          - type: ndcg_at_1000
            value: 32.303
          - type: ndcg_at_3
            value: 21.719
          - type: ndcg_at_5
            value: 23.039
          - type: precision_at_1
            value: 18.865000000000002
          - type: precision_at_10
            value: 4.263999999999999
          - type: precision_at_100
            value: 0.696
          - type: precision_at_1000
            value: 0.1
          - type: precision_at_3
            value: 9.866999999999999
          - type: precision_at_5
            value: 6.902
          - type: recall_at_1
            value: 16.07
          - type: recall_at_10
            value: 33.661
          - type: recall_at_100
            value: 54.001999999999995
          - type: recall_at_1000
            value: 75.564
          - type: recall_at_3
            value: 23.956
          - type: recall_at_5
            value: 27.264
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackTexRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 10.847
          - type: map_at_10
            value: 15.518
          - type: map_at_100
            value: 16.384
          - type: map_at_1000
            value: 16.506
          - type: map_at_3
            value: 14.093
          - type: map_at_5
            value: 14.868
          - type: ndcg_at_1
            value: 13.764999999999999
          - type: ndcg_at_10
            value: 18.766
          - type: ndcg_at_100
            value: 23.076
          - type: ndcg_at_1000
            value: 26.344
          - type: ndcg_at_3
            value: 16.150000000000002
          - type: ndcg_at_5
            value: 17.373
          - type: precision_at_1
            value: 13.764999999999999
          - type: precision_at_10
            value: 3.572
          - type: precision_at_100
            value: 0.6779999999999999
          - type: precision_at_1000
            value: 0.11199999999999999
          - type: precision_at_3
            value: 7.88
          - type: precision_at_5
            value: 5.712
          - type: recall_at_1
            value: 10.847
          - type: recall_at_10
            value: 25.141999999999996
          - type: recall_at_100
            value: 44.847
          - type: recall_at_1000
            value: 68.92099999999999
          - type: recall_at_3
            value: 17.721999999999998
          - type: recall_at_5
            value: 20.968999999999998
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackUnixRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 18.377
          - type: map_at_10
            value: 26.005
          - type: map_at_100
            value: 26.996
          - type: map_at_1000
            value: 27.116
          - type: map_at_3
            value: 23.712
          - type: map_at_5
            value: 24.859
          - type: ndcg_at_1
            value: 22.201
          - type: ndcg_at_10
            value: 30.635
          - type: ndcg_at_100
            value: 35.623
          - type: ndcg_at_1000
            value: 38.551
          - type: ndcg_at_3
            value: 26.565
          - type: ndcg_at_5
            value: 28.28
          - type: precision_at_1
            value: 22.201
          - type: precision_at_10
            value: 5.41
          - type: precision_at_100
            value: 0.88
          - type: precision_at_1000
            value: 0.125
          - type: precision_at_3
            value: 12.531
          - type: precision_at_5
            value: 8.806
          - type: recall_at_1
            value: 18.377
          - type: recall_at_10
            value: 40.908
          - type: recall_at_100
            value: 63.563
          - type: recall_at_1000
            value: 84.503
          - type: recall_at_3
            value: 29.793999999999997
          - type: recall_at_5
            value: 34.144999999999996
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackWebmastersRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 20.246
          - type: map_at_10
            value: 27.528000000000002
          - type: map_at_100
            value: 28.78
          - type: map_at_1000
            value: 29.002
          - type: map_at_3
            value: 25.226
          - type: map_at_5
            value: 26.355
          - type: ndcg_at_1
            value: 25.099
          - type: ndcg_at_10
            value: 32.421
          - type: ndcg_at_100
            value: 37.2
          - type: ndcg_at_1000
            value: 40.693
          - type: ndcg_at_3
            value: 28.768
          - type: ndcg_at_5
            value: 30.23
          - type: precision_at_1
            value: 25.099
          - type: precision_at_10
            value: 6.245
          - type: precision_at_100
            value: 1.269
          - type: precision_at_1000
            value: 0.218
          - type: precision_at_3
            value: 13.767999999999999
          - type: precision_at_5
            value: 9.881
          - type: recall_at_1
            value: 20.246
          - type: recall_at_10
            value: 41.336
          - type: recall_at_100
            value: 63.098
          - type: recall_at_1000
            value: 86.473
          - type: recall_at_3
            value: 30.069000000000003
          - type: recall_at_5
            value: 34.262
      - task:
          type: Retrieval
        dataset:
          type: BeIR/cqadupstack
          name: MTEB CQADupstackWordpressRetrieval
          config: default
          split: test
          revision: 2b9f5791698b5be7bc5e10535c8690f20043c3db
        metrics:
          - type: map_at_1
            value: 14.054
          - type: map_at_10
            value: 20.25
          - type: map_at_100
            value: 21.178
          - type: map_at_1000
            value: 21.288999999999998
          - type: map_at_3
            value: 18.584999999999997
          - type: map_at_5
            value: 19.536
          - type: ndcg_at_1
            value: 15.527
          - type: ndcg_at_10
            value: 23.745
          - type: ndcg_at_100
            value: 28.610999999999997
          - type: ndcg_at_1000
            value: 31.740000000000002
          - type: ndcg_at_3
            value: 20.461
          - type: ndcg_at_5
            value: 22.072
          - type: precision_at_1
            value: 15.527
          - type: precision_at_10
            value: 3.882
          - type: precision_at_100
            value: 0.6930000000000001
          - type: precision_at_1000
            value: 0.104
          - type: precision_at_3
            value: 9.181000000000001
          - type: precision_at_5
            value: 6.433
          - type: recall_at_1
            value: 14.054
          - type: recall_at_10
            value: 32.714
          - type: recall_at_100
            value: 55.723
          - type: recall_at_1000
            value: 79.72399999999999
          - type: recall_at_3
            value: 23.832
          - type: recall_at_5
            value: 27.754
      - task:
          type: Retrieval
        dataset:
          type: climate-fever
          name: MTEB ClimateFEVER
          config: default
          split: test
          revision: 392b78eb68c07badcd7c2cd8f39af108375dfcce
        metrics:
          - type: map_at_1
            value: 6.122
          - type: map_at_10
            value: 11.556
          - type: map_at_100
            value: 12.998000000000001
          - type: map_at_1000
            value: 13.202
          - type: map_at_3
            value: 9.657
          - type: map_at_5
            value: 10.585
          - type: ndcg_at_1
            value: 15.049000000000001
          - type: ndcg_at_10
            value: 17.574
          - type: ndcg_at_100
            value: 24.465999999999998
          - type: ndcg_at_1000
            value: 28.511999999999997
          - type: ndcg_at_3
            value: 13.931
          - type: ndcg_at_5
            value: 15.112
          - type: precision_at_1
            value: 15.049000000000001
          - type: precision_at_10
            value: 5.831
          - type: precision_at_100
            value: 1.322
          - type: precision_at_1000
            value: 0.20500000000000002
          - type: precision_at_3
            value: 10.749
          - type: precision_at_5
            value: 8.365
          - type: recall_at_1
            value: 6.122
          - type: recall_at_10
            value: 22.207
          - type: recall_at_100
            value: 47.08
          - type: recall_at_1000
            value: 70.182
          - type: recall_at_3
            value: 13.416
          - type: recall_at_5
            value: 16.672
      - task:
          type: Retrieval
        dataset:
          type: dbpedia-entity
          name: MTEB DBPedia
          config: default
          split: test
          revision: f097057d03ed98220bc7309ddb10b71a54d667d6
        metrics:
          - type: map_at_1
            value: 4.672
          - type: map_at_10
            value: 10.534
          - type: map_at_100
            value: 14.798
          - type: map_at_1000
            value: 15.927
          - type: map_at_3
            value: 7.317
          - type: map_at_5
            value: 8.726
          - type: ndcg_at_1
            value: 36.5
          - type: ndcg_at_10
            value: 26.098
          - type: ndcg_at_100
            value: 29.215999999999998
          - type: ndcg_at_1000
            value: 36.254999999999995
          - type: ndcg_at_3
            value: 29.247
          - type: ndcg_at_5
            value: 27.692
          - type: precision_at_1
            value: 47.25
          - type: precision_at_10
            value: 22.625
          - type: precision_at_100
            value: 7.042
          - type: precision_at_1000
            value: 1.6129999999999998
          - type: precision_at_3
            value: 34.083000000000006
          - type: precision_at_5
            value: 29.5
          - type: recall_at_1
            value: 4.672
          - type: recall_at_10
            value: 15.638
          - type: recall_at_100
            value: 36.228
          - type: recall_at_1000
            value: 58.831
          - type: recall_at_3
            value: 8.578
          - type: recall_at_5
            value: 11.18
      - task:
          type: Classification
        dataset:
          type: mteb/emotion
          name: MTEB EmotionClassification
          config: default
          split: test
          revision: 829147f8f75a25f005913200eb5ed41fae320aa1
        metrics:
          - type: accuracy
            value: 49.919999999999995
          - type: f1
            value: 45.37973678791632
      - task:
          type: Retrieval
        dataset:
          type: fever
          name: MTEB FEVER
          config: default
          split: test
          revision: 1429cf27e393599b8b359b9b72c666f96b2525f9
        metrics:
          - type: map_at_1
            value: 25.801000000000002
          - type: map_at_10
            value: 33.941
          - type: map_at_100
            value: 34.73
          - type: map_at_1000
            value: 34.793
          - type: map_at_3
            value: 31.705
          - type: map_at_5
            value: 33.047
          - type: ndcg_at_1
            value: 27.933000000000003
          - type: ndcg_at_10
            value: 38.644
          - type: ndcg_at_100
            value: 42.594
          - type: ndcg_at_1000
            value: 44.352000000000004
          - type: ndcg_at_3
            value: 34.199
          - type: ndcg_at_5
            value: 36.573
          - type: precision_at_1
            value: 27.933000000000003
          - type: precision_at_10
            value: 5.603000000000001
          - type: precision_at_100
            value: 0.773
          - type: precision_at_1000
            value: 0.094
          - type: precision_at_3
            value: 14.171
          - type: precision_at_5
            value: 9.786999999999999
          - type: recall_at_1
            value: 25.801000000000002
          - type: recall_at_10
            value: 50.876
          - type: recall_at_100
            value: 69.253
          - type: recall_at_1000
            value: 82.907
          - type: recall_at_3
            value: 38.879000000000005
          - type: recall_at_5
            value: 44.651999999999994
      - task:
          type: Retrieval
        dataset:
          type: fiqa
          name: MTEB FiQA2018
          config: default
          split: test
          revision: 41b686a7f28c59bcaaa5791efd47c67c8ebe28be
        metrics:
          - type: map_at_1
            value: 9.142
          - type: map_at_10
            value: 13.841999999999999
          - type: map_at_100
            value: 14.960999999999999
          - type: map_at_1000
            value: 15.187000000000001
          - type: map_at_3
            value: 11.966000000000001
          - type: map_at_5
            value: 12.921
          - type: ndcg_at_1
            value: 18.364
          - type: ndcg_at_10
            value: 18.590999999999998
          - type: ndcg_at_100
            value: 24.153
          - type: ndcg_at_1000
            value: 29.104000000000003
          - type: ndcg_at_3
            value: 16.323
          - type: ndcg_at_5
            value: 17.000999999999998
          - type: precision_at_1
            value: 18.364
          - type: precision_at_10
            value: 5.216
          - type: precision_at_100
            value: 1.09
          - type: precision_at_1000
            value: 0.193
          - type: precision_at_3
            value: 10.751
          - type: precision_at_5
            value: 7.932
          - type: recall_at_1
            value: 9.142
          - type: recall_at_10
            value: 22.747
          - type: recall_at_100
            value: 44.585
          - type: recall_at_1000
            value: 75.481
          - type: recall_at_3
            value: 14.602
          - type: recall_at_5
            value: 17.957
      - task:
          type: Retrieval
        dataset:
          type: hotpotqa
          name: MTEB HotpotQA
          config: default
          split: test
          revision: 766870b35a1b9ca65e67a0d1913899973551fc6c
        metrics:
          - type: map_at_1
            value: 18.677
          - type: map_at_10
            value: 26.616
          - type: map_at_100
            value: 27.605
          - type: map_at_1000
            value: 27.711999999999996
          - type: map_at_3
            value: 24.396
          - type: map_at_5
            value: 25.627
          - type: ndcg_at_1
            value: 37.352999999999994
          - type: ndcg_at_10
            value: 33.995
          - type: ndcg_at_100
            value: 38.423
          - type: ndcg_at_1000
            value: 40.947
          - type: ndcg_at_3
            value: 29.885
          - type: ndcg_at_5
            value: 31.874999999999996
          - type: precision_at_1
            value: 37.352999999999994
          - type: precision_at_10
            value: 7.539999999999999
          - type: precision_at_100
            value: 1.107
          - type: precision_at_1000
            value: 0.145
          - type: precision_at_3
            value: 18.938
          - type: precision_at_5
            value: 12.943
          - type: recall_at_1
            value: 18.677
          - type: recall_at_10
            value: 37.698
          - type: recall_at_100
            value: 55.354000000000006
          - type: recall_at_1000
            value: 72.255
          - type: recall_at_3
            value: 28.406
          - type: recall_at_5
            value: 32.357
      - task:
          type: Classification
        dataset:
          type: mteb/imdb
          name: MTEB ImdbClassification
          config: default
          split: test
          revision: 8d743909f834c38949e8323a8a6ce8721ea6c7f4
        metrics:
          - type: accuracy
            value: 74.3292
          - type: ap
            value: 68.30186110189658
          - type: f1
            value: 74.20709636944783
      - task:
          type: Retrieval
        dataset:
          type: msmarco
          name: MTEB MSMARCO
          config: default
          split: validation
          revision: e6838a846e2408f22cf5cc337ebc83e0bcf77849
        metrics:
          - type: map_at_1
            value: 6.889000000000001
          - type: map_at_10
            value: 12.321
          - type: map_at_100
            value: 13.416
          - type: map_at_1000
            value: 13.525
          - type: map_at_3
            value: 10.205
          - type: map_at_5
            value: 11.342
          - type: ndcg_at_1
            value: 7.092
          - type: ndcg_at_10
            value: 15.827
          - type: ndcg_at_100
            value: 21.72
          - type: ndcg_at_1000
            value: 24.836
          - type: ndcg_at_3
            value: 11.393
          - type: ndcg_at_5
            value: 13.462
          - type: precision_at_1
            value: 7.092
          - type: precision_at_10
            value: 2.7969999999999997
          - type: precision_at_100
            value: 0.583
          - type: precision_at_1000
            value: 0.08499999999999999
          - type: precision_at_3
            value: 5.019
          - type: precision_at_5
            value: 4.06
          - type: recall_at_1
            value: 6.889000000000001
          - type: recall_at_10
            value: 26.791999999999998
          - type: recall_at_100
            value: 55.371
          - type: recall_at_1000
            value: 80.12899999999999
          - type: recall_at_3
            value: 14.573
          - type: recall_at_5
            value: 19.557
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_domain
          name: MTEB MTOPDomainClassification (en)
          config: en
          split: test
          revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
        metrics:
          - type: accuracy
            value: 89.6374829001368
          - type: f1
            value: 89.20878379358307
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_domain
          name: MTEB MTOPDomainClassification (de)
          config: de
          split: test
          revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
        metrics:
          - type: accuracy
            value: 84.54212454212454
          - type: f1
            value: 82.81080100037023
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_domain
          name: MTEB MTOPDomainClassification (es)
          config: es
          split: test
          revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
        metrics:
          - type: accuracy
            value: 86.46430953969313
          - type: f1
            value: 86.00019824223267
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_domain
          name: MTEB MTOPDomainClassification (fr)
          config: fr
          split: test
          revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
        metrics:
          - type: accuracy
            value: 81.31850923896022
          - type: f1
            value: 81.07860454762863
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_domain
          name: MTEB MTOPDomainClassification (hi)
          config: hi
          split: test
          revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
        metrics:
          - type: accuracy
            value: 58.23234134098243
          - type: f1
            value: 56.63845098081841
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_domain
          name: MTEB MTOPDomainClassification (th)
          config: th
          split: test
          revision: a7e2a951126a26fc8c6a69f835f33a346ba259e3
        metrics:
          - type: accuracy
            value: 72.28571428571429
          - type: f1
            value: 70.95796714592039
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_intent
          name: MTEB MTOPIntentClassification (en)
          config: en
          split: test
          revision: 6299947a7777084cc2d4b64235bf7190381ce755
        metrics:
          - type: accuracy
            value: 70.68171454628363
          - type: f1
            value: 52.57188062729139
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_intent
          name: MTEB MTOPIntentClassification (de)
          config: de
          split: test
          revision: 6299947a7777084cc2d4b64235bf7190381ce755
        metrics:
          - type: accuracy
            value: 60.521273598196665
          - type: f1
            value: 42.70492970339204
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_intent
          name: MTEB MTOPIntentClassification (es)
          config: es
          split: test
          revision: 6299947a7777084cc2d4b64235bf7190381ce755
        metrics:
          - type: accuracy
            value: 64.32288192128087
          - type: f1
            value: 45.97360620220273
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_intent
          name: MTEB MTOPIntentClassification (fr)
          config: fr
          split: test
          revision: 6299947a7777084cc2d4b64235bf7190381ce755
        metrics:
          - type: accuracy
            value: 58.67209520826808
          - type: f1
            value: 42.82844991304579
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_intent
          name: MTEB MTOPIntentClassification (hi)
          config: hi
          split: test
          revision: 6299947a7777084cc2d4b64235bf7190381ce755
        metrics:
          - type: accuracy
            value: 41.95769092864826
          - type: f1
            value: 28.914127631431263
      - task:
          type: Classification
        dataset:
          type: mteb/mtop_intent
          name: MTEB MTOPIntentClassification (th)
          config: th
          split: test
          revision: 6299947a7777084cc2d4b64235bf7190381ce755
        metrics:
          - type: accuracy
            value: 55.28390596745027
          - type: f1
            value: 38.33899250561289
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_intent
          name: MTEB MassiveIntentClassification (en)
          config: en
          split: test
          revision: 072a486a144adf7f4479a4a0dddb2152e161e1ea
        metrics:
          - type: accuracy
            value: 70.00336247478144
          - type: f1
            value: 68.72041942191649
      - task:
          type: Classification
        dataset:
          type: mteb/amazon_massive_scenario
          name: MTEB MassiveScenarioClassification (en)
          config: en
          split: test
          revision: 7d571f92784cd94a019292a1f45445077d0ef634
        metrics:
          - type: accuracy
            value: 75.0268997982515
          - type: f1
            value: 75.29844481506652
      - task:
          type: Clustering
        dataset:
          type: mteb/medrxiv-clustering-p2p
          name: MTEB MedrxivClusteringP2P
          config: default
          split: test
          revision: dcefc037ef84348e49b0d29109e891c01067226b
        metrics:
          - type: v_measure
            value: 30.327566856300813
      - task:
          type: Clustering
        dataset:
          type: mteb/medrxiv-clustering-s2s
          name: MTEB MedrxivClusteringS2S
          config: default
          split: test
          revision: 3cd0e71dfbe09d4de0f9e5ecba43e7ce280959dc
        metrics:
          - type: v_measure
            value: 28.01650210863619
      - task:
          type: Reranking
        dataset:
          type: mteb/mind_small
          name: MTEB MindSmallReranking
          config: default
          split: test
          revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
        metrics:
          - type: map
            value: 31.11041256752524
          - type: mrr
            value: 32.14172939750204
      - task:
          type: Retrieval
        dataset:
          type: nfcorpus
          name: MTEB NFCorpus
          config: default
          split: test
          revision: 7eb63cc0c1eb59324d709ebed25fcab851fa7610
        metrics:
          - type: map_at_1
            value: 3.527
          - type: map_at_10
            value: 9.283
          - type: map_at_100
            value: 11.995000000000001
          - type: map_at_1000
            value: 13.33
          - type: map_at_3
            value: 6.223
          - type: map_at_5
            value: 7.68
          - type: ndcg_at_1
            value: 36.223
          - type: ndcg_at_10
            value: 28.255999999999997
          - type: ndcg_at_100
            value: 26.355
          - type: ndcg_at_1000
            value: 35.536
          - type: ndcg_at_3
            value: 31.962000000000003
          - type: ndcg_at_5
            value: 30.61
          - type: precision_at_1
            value: 37.771
          - type: precision_at_10
            value: 21.889
          - type: precision_at_100
            value: 7.1080000000000005
          - type: precision_at_1000
            value: 1.989
          - type: precision_at_3
            value: 30.857
          - type: precision_at_5
            value: 27.307
          - type: recall_at_1
            value: 3.527
          - type: recall_at_10
            value: 14.015
          - type: recall_at_100
            value: 28.402
          - type: recall_at_1000
            value: 59.795
          - type: recall_at_3
            value: 7.5969999999999995
          - type: recall_at_5
            value: 10.641
      - task:
          type: Retrieval
        dataset:
          type: nq
          name: MTEB NQ
          config: default
          split: test
          revision: 6062aefc120bfe8ece5897809fb2e53bfe0d128c
        metrics:
          - type: map_at_1
            value: 11.631
          - type: map_at_10
            value: 19.532
          - type: map_at_100
            value: 20.821
          - type: map_at_1000
            value: 20.910999999999998
          - type: map_at_3
            value: 16.597
          - type: map_at_5
            value: 18.197
          - type: ndcg_at_1
            value: 13.413
          - type: ndcg_at_10
            value: 24.628
          - type: ndcg_at_100
            value: 30.883
          - type: ndcg_at_1000
            value: 33.216
          - type: ndcg_at_3
            value: 18.697
          - type: ndcg_at_5
            value: 21.501
          - type: precision_at_1
            value: 13.413
          - type: precision_at_10
            value: 4.571
          - type: precision_at_100
            value: 0.812
          - type: precision_at_1000
            value: 0.10300000000000001
          - type: precision_at_3
            value: 8.845
          - type: precision_at_5
            value: 6.889000000000001
          - type: recall_at_1
            value: 11.631
          - type: recall_at_10
            value: 38.429
          - type: recall_at_100
            value: 67.009
          - type: recall_at_1000
            value: 84.796
          - type: recall_at_3
            value: 22.74
          - type: recall_at_5
            value: 29.266
      - task:
          type: Retrieval
        dataset:
          type: quora
          name: MTEB QuoraRetrieval
          config: default
          split: test
          revision: 6205996560df11e3a3da9ab4f926788fc30a7db4
        metrics:
          - type: map_at_1
            value: 66.64
          - type: map_at_10
            value: 80.394
          - type: map_at_100
            value: 81.099
          - type: map_at_1000
            value: 81.122
          - type: map_at_3
            value: 77.289
          - type: map_at_5
            value: 79.25999999999999
          - type: ndcg_at_1
            value: 76.85
          - type: ndcg_at_10
            value: 84.68
          - type: ndcg_at_100
            value: 86.311
          - type: ndcg_at_1000
            value: 86.49900000000001
          - type: ndcg_at_3
            value: 81.295
          - type: ndcg_at_5
            value: 83.199
          - type: precision_at_1
            value: 76.85
          - type: precision_at_10
            value: 12.928999999999998
          - type: precision_at_100
            value: 1.51
          - type: precision_at_1000
            value: 0.156
          - type: precision_at_3
            value: 35.557
          - type: precision_at_5
            value: 23.576
          - type: recall_at_1
            value: 66.64
          - type: recall_at_10
            value: 93.059
          - type: recall_at_100
            value: 98.922
          - type: recall_at_1000
            value: 99.883
          - type: recall_at_3
            value: 83.49499999999999
          - type: recall_at_5
            value: 88.729
      - task:
          type: Clustering
        dataset:
          type: mteb/reddit-clustering
          name: MTEB RedditClustering
          config: default
          split: test
          revision: b2805658ae38990172679479369a78b86de8c390
        metrics:
          - type: v_measure
            value: 42.17131361041068
      - task:
          type: Clustering
        dataset:
          type: mteb/reddit-clustering-p2p
          name: MTEB RedditClusteringP2P
          config: default
          split: test
          revision: 385e3cb46b4cfa89021f56c4380204149d0efe33
        metrics:
          - type: v_measure
            value: 48.01815621479994
      - task:
          type: Retrieval
        dataset:
          type: scidocs
          name: MTEB SCIDOCS
          config: default
          split: test
          revision: 5c59ef3e437a0a9651c8fe6fde943e7dce59fba5
        metrics:
          - type: map_at_1
            value: 3.198
          - type: map_at_10
            value: 7.550999999999999
          - type: map_at_100
            value: 9.232
          - type: map_at_1000
            value: 9.51
          - type: map_at_3
            value: 5.2940000000000005
          - type: map_at_5
            value: 6.343999999999999
          - type: ndcg_at_1
            value: 15.8
          - type: ndcg_at_10
            value: 13.553999999999998
          - type: ndcg_at_100
            value: 20.776
          - type: ndcg_at_1000
            value: 26.204
          - type: ndcg_at_3
            value: 12.306000000000001
          - type: ndcg_at_5
            value: 10.952
          - type: precision_at_1
            value: 15.8
          - type: precision_at_10
            value: 7.180000000000001
          - type: precision_at_100
            value: 1.762
          - type: precision_at_1000
            value: 0.307
          - type: precision_at_3
            value: 11.333
          - type: precision_at_5
            value: 9.62
          - type: recall_at_1
            value: 3.198
          - type: recall_at_10
            value: 14.575
          - type: recall_at_100
            value: 35.758
          - type: recall_at_1000
            value: 62.317
          - type: recall_at_3
            value: 6.922000000000001
          - type: recall_at_5
            value: 9.767000000000001
      - task:
          type: STS
        dataset:
          type: mteb/sickr-sts
          name: MTEB SICK-R
          config: default
          split: test
          revision: 20a6d6f312dd54037fe07a32d58e5e168867909d
        metrics:
          - type: cos_sim_pearson
            value: 84.5217161312271
          - type: cos_sim_spearman
            value: 79.58562467776268
          - type: euclidean_pearson
            value: 76.69364353942403
          - type: euclidean_spearman
            value: 74.68959282070473
          - type: manhattan_pearson
            value: 76.81159265133732
          - type: manhattan_spearman
            value: 74.7519444048176
      - task:
          type: STS
        dataset:
          type: mteb/sts12-sts
          name: MTEB STS12
          config: default
          split: test
          revision: fdf84275bb8ce4b49c971d02e84dd1abc677a50f
        metrics:
          - type: cos_sim_pearson
            value: 83.70403706922605
          - type: cos_sim_spearman
            value: 74.28502198729447
          - type: euclidean_pearson
            value: 83.32719404608066
          - type: euclidean_spearman
            value: 75.92189433460788
          - type: manhattan_pearson
            value: 83.35841543005293
          - type: manhattan_spearman
            value: 75.94458615451978
      - task:
          type: STS
        dataset:
          type: mteb/sts13-sts
          name: MTEB STS13
          config: default
          split: test
          revision: 1591bfcbe8c69d4bf7fe2a16e2451017832cafb9
        metrics:
          - type: cos_sim_pearson
            value: 84.94127878986795
          - type: cos_sim_spearman
            value: 85.35148434923192
          - type: euclidean_pearson
            value: 81.71127467071571
          - type: euclidean_spearman
            value: 82.88240481546771
          - type: manhattan_pearson
            value: 81.72826221967252
          - type: manhattan_spearman
            value: 82.90725064625128
      - task:
          type: STS
        dataset:
          type: mteb/sts14-sts
          name: MTEB STS14
          config: default
          split: test
          revision: e2125984e7df8b7871f6ae9949cf6b6795e7c54b
        metrics:
          - type: cos_sim_pearson
            value: 83.1474704168523
          - type: cos_sim_spearman
            value: 79.20612995350827
          - type: euclidean_pearson
            value: 78.85993329596555
          - type: euclidean_spearman
            value: 78.91956572744715
          - type: manhattan_pearson
            value: 78.89999720522347
          - type: manhattan_spearman
            value: 78.93956842550107
      - task:
          type: STS
        dataset:
          type: mteb/sts15-sts
          name: MTEB STS15
          config: default
          split: test
          revision: 1cd7298cac12a96a373b6a2f18738bb3e739a9b6
        metrics:
          - type: cos_sim_pearson
            value: 84.81255514055894
          - type: cos_sim_spearman
            value: 85.5217140762934
          - type: euclidean_pearson
            value: 82.15024353784499
          - type: euclidean_spearman
            value: 83.04155334389833
          - type: manhattan_pearson
            value: 82.18598945053624
          - type: manhattan_spearman
            value: 83.07248357693301
      - task:
          type: STS
        dataset:
          type: mteb/sts16-sts
          name: MTEB STS16
          config: default
          split: test
          revision: 360a0b2dff98700d09e634a01e1cc1624d3e42cd
        metrics:
          - type: cos_sim_pearson
            value: 80.63248465157822
          - type: cos_sim_spearman
            value: 82.53853238521991
          - type: euclidean_pearson
            value: 78.33936863828221
          - type: euclidean_spearman
            value: 79.16305579487414
          - type: manhattan_pearson
            value: 78.3888359870894
          - type: manhattan_spearman
            value: 79.18504473136467
      - task:
          type: STS
        dataset:
          type: mteb/sts17-crosslingual-sts
          name: MTEB STS17 (en-en)
          config: en-en
          split: test
          revision: 9fc37e8c632af1c87a3d23e685d49552a02582a0
        metrics:
          - type: cos_sim_pearson
            value: 90.09066290639687
          - type: cos_sim_spearman
            value: 90.43893699357069
          - type: euclidean_pearson
            value: 82.39520777222396
          - type: euclidean_spearman
            value: 81.23948185395952
          - type: manhattan_pearson
            value: 82.35529784653383
          - type: manhattan_spearman
            value: 81.12681522483975
      - task:
          type: STS
        dataset:
          type: mteb/sts22-crosslingual-sts
          name: MTEB STS22 (en)
          config: en
          split: test
          revision: 2de6ce8c1921b71a755b262c6b57fef195dd7906
        metrics:
          - type: cos_sim_pearson
            value: 63.52752323046846
          - type: cos_sim_spearman
            value: 63.19719780439462
          - type: euclidean_pearson
            value: 58.29085490641428
          - type: euclidean_spearman
            value: 58.975178656335046
          - type: manhattan_pearson
            value: 58.183542772416985
          - type: manhattan_spearman
            value: 59.190630462178994
      - task:
          type: STS
        dataset:
          type: mteb/stsbenchmark-sts
          name: MTEB STSBenchmark
          config: default
          split: test
          revision: 8913289635987208e6e7c72789e4be2fe94b6abd
        metrics:
          - type: cos_sim_pearson
            value: 85.45100366635687
          - type: cos_sim_spearman
            value: 85.66816193002651
          - type: euclidean_pearson
            value: 81.87976731329091
          - type: euclidean_spearman
            value: 82.01382867690964
          - type: manhattan_pearson
            value: 81.88260155706726
          - type: manhattan_spearman
            value: 82.05258597906492
      - task:
          type: Reranking
        dataset:
          type: mteb/scidocs-reranking
          name: MTEB SciDocsRR
          config: default
          split: test
          revision: 56a6d0140cf6356659e2a7c1413286a774468d44
        metrics:
          - type: map
            value: 77.53549990038017
          - type: mrr
            value: 93.37474163454556
      - task:
          type: Retrieval
        dataset:
          type: scifact
          name: MTEB SciFact
          config: default
          split: test
          revision: a75ae049398addde9b70f6b268875f5cbce99089
        metrics:
          - type: map_at_1
            value: 31.167
          - type: map_at_10
            value: 40.778
          - type: map_at_100
            value: 42.063
          - type: map_at_1000
            value: 42.103
          - type: map_at_3
            value: 37.12
          - type: map_at_5
            value: 39.205
          - type: ndcg_at_1
            value: 33.667
          - type: ndcg_at_10
            value: 46.662
          - type: ndcg_at_100
            value: 51.995999999999995
          - type: ndcg_at_1000
            value: 53.254999999999995
          - type: ndcg_at_3
            value: 39.397999999999996
          - type: ndcg_at_5
            value: 42.934
          - type: precision_at_1
            value: 33.667
          - type: precision_at_10
            value: 7.1
          - type: precision_at_100
            value: 0.993
          - type: precision_at_1000
            value: 0.11
          - type: precision_at_3
            value: 16.111
          - type: precision_at_5
            value: 11.600000000000001
          - type: recall_at_1
            value: 31.167
          - type: recall_at_10
            value: 63.744
          - type: recall_at_100
            value: 87.156
          - type: recall_at_1000
            value: 97.556
          - type: recall_at_3
            value: 44
          - type: recall_at_5
            value: 52.556000000000004
      - task:
          type: PairClassification
        dataset:
          type: mteb/sprintduplicatequestions-pairclassification
          name: MTEB SprintDuplicateQuestions
          config: default
          split: test
          revision: 5a8256d0dff9c4bd3be3ba3e67e4e70173f802ea
        metrics:
          - type: cos_sim_accuracy
            value: 99.55148514851486
          - type: cos_sim_ap
            value: 80.535236573428
          - type: cos_sim_f1
            value: 75.01331912626532
          - type: cos_sim_precision
            value: 80.27366020524515
          - type: cos_sim_recall
            value: 70.39999999999999
          - type: dot_accuracy
            value: 99.04851485148515
          - type: dot_ap
            value: 28.505358821499726
          - type: dot_f1
            value: 36.36363636363637
          - type: dot_precision
            value: 37.160751565762006
          - type: dot_recall
            value: 35.6
          - type: euclidean_accuracy
            value: 99.4990099009901
          - type: euclidean_ap
            value: 74.95819047075476
          - type: euclidean_f1
            value: 71.15489874110564
          - type: euclidean_precision
            value: 78.59733978234583
          - type: euclidean_recall
            value: 65
          - type: manhattan_accuracy
            value: 99.50198019801981
          - type: manhattan_ap
            value: 75.02070096015086
          - type: manhattan_f1
            value: 71.20535714285712
          - type: manhattan_precision
            value: 80.55555555555556
          - type: manhattan_recall
            value: 63.800000000000004
          - type: max_accuracy
            value: 99.55148514851486
          - type: max_ap
            value: 80.535236573428
          - type: max_f1
            value: 75.01331912626532
      - task:
          type: Clustering
        dataset:
          type: mteb/stackexchange-clustering
          name: MTEB StackExchangeClustering
          config: default
          split: test
          revision: 70a89468f6dccacc6aa2b12a6eac54e74328f235
        metrics:
          - type: v_measure
            value: 54.13314692311623
      - task:
          type: Clustering
        dataset:
          type: mteb/stackexchange-clustering-p2p
          name: MTEB StackExchangeClusteringP2P
          config: default
          split: test
          revision: d88009ab563dd0b16cfaf4436abaf97fa3550cf0
        metrics:
          - type: v_measure
            value: 31.115181648287145
      - task:
          type: Reranking
        dataset:
          type: mteb/stackoverflowdupquestions-reranking
          name: MTEB StackOverflowDupQuestions
          config: default
          split: test
          revision: ef807ea29a75ec4f91b50fd4191cb4ee4589a9f9
        metrics:
          - type: map
            value: 44.771112666694336
          - type: mrr
            value: 45.30415764790765
      - task:
          type: Summarization
        dataset:
          type: mteb/summeval
          name: MTEB SummEval
          config: default
          split: test
          revision: 8753c2788d36c01fc6f05d03fe3f7268d63f9122
        metrics:
          - type: cos_sim_pearson
            value: 30.849429597669374
          - type: cos_sim_spearman
            value: 30.384175038360194
          - type: dot_pearson
            value: 29.030383429536823
          - type: dot_spearman
            value: 28.03273624951732
      - task:
          type: Retrieval
        dataset:
          type: trec-covid
          name: MTEB TRECCOVID
          config: default
          split: test
          revision: 2c8041b2c07a79b6f7ba8fe6acc72e5d9f92d217
        metrics:
          - type: map_at_1
            value: 0.19499999999999998
          - type: map_at_10
            value: 1.0959999999999999
          - type: map_at_100
            value: 5.726
          - type: map_at_1000
            value: 13.611999999999998
          - type: map_at_3
            value: 0.45399999999999996
          - type: map_at_5
            value: 0.67
          - type: ndcg_at_1
            value: 71
          - type: ndcg_at_10
            value: 55.352999999999994
          - type: ndcg_at_100
            value: 40.797
          - type: ndcg_at_1000
            value: 35.955999999999996
          - type: ndcg_at_3
            value: 63.263000000000005
          - type: ndcg_at_5
            value: 60.14000000000001
          - type: precision_at_1
            value: 78
          - type: precision_at_10
            value: 56.99999999999999
          - type: precision_at_100
            value: 41.199999999999996
          - type: precision_at_1000
            value: 16.154
          - type: precision_at_3
            value: 66.667
          - type: precision_at_5
            value: 62.8
          - type: recall_at_1
            value: 0.19499999999999998
          - type: recall_at_10
            value: 1.3639999999999999
          - type: recall_at_100
            value: 9.317
          - type: recall_at_1000
            value: 33.629999999999995
          - type: recall_at_3
            value: 0.49300000000000005
          - type: recall_at_5
            value: 0.756
      - task:
          type: Retrieval
        dataset:
          type: webis-touche2020
          name: MTEB Touche2020
          config: default
          split: test
          revision: 527b7d77e16e343303e68cb6af11d6e18b9f7b3b
        metrics:
          - type: map_at_1
            value: 1.335
          - type: map_at_10
            value: 6.293
          - type: map_at_100
            value: 10.928
          - type: map_at_1000
            value: 12.359
          - type: map_at_3
            value: 3.472
          - type: map_at_5
            value: 4.935
          - type: ndcg_at_1
            value: 19.387999999999998
          - type: ndcg_at_10
            value: 16.178
          - type: ndcg_at_100
            value: 28.149
          - type: ndcg_at_1000
            value: 39.845000000000006
          - type: ndcg_at_3
            value: 19.171
          - type: ndcg_at_5
            value: 17.864
          - type: precision_at_1
            value: 20.408
          - type: precision_at_10
            value: 14.49
          - type: precision_at_100
            value: 6.306000000000001
          - type: precision_at_1000
            value: 1.3860000000000001
          - type: precision_at_3
            value: 21.088
          - type: precision_at_5
            value: 18.367
          - type: recall_at_1
            value: 1.335
          - type: recall_at_10
            value: 10.825999999999999
          - type: recall_at_100
            value: 39.251000000000005
          - type: recall_at_1000
            value: 74.952
          - type: recall_at_3
            value: 4.9110000000000005
          - type: recall_at_5
            value: 7.312
      - task:
          type: Classification
        dataset:
          type: mteb/toxic_conversations_50k
          name: MTEB ToxicConversationsClassification
          config: default
          split: test
          revision: edfaf9da55d3dd50d43143d90c1ac476895ae6de
        metrics:
          - type: accuracy
            value: 69.93339999999999
          - type: ap
            value: 13.87476602492533
          - type: f1
            value: 53.867357615848555
      - task:
          type: Classification
        dataset:
          type: mteb/tweet_sentiment_extraction
          name: MTEB TweetSentimentExtractionClassification
          config: default
          split: test
          revision: 62146448f05be9e52a36b8ee9936447ea787eede
        metrics:
          - type: accuracy
            value: 62.43916242218449
          - type: f1
            value: 62.870386304954685
      - task:
          type: Clustering
        dataset:
          type: mteb/twentynewsgroups-clustering
          name: MTEB TwentyNewsgroupsClustering
          config: default
          split: test
          revision: 091a54f9a36281ce7d6590ec8c75dd485e7e01d4
        metrics:
          - type: v_measure
            value: 37.202082549859796
      - task:
          type: PairClassification
        dataset:
          type: mteb/twittersemeval2015-pairclassification
          name: MTEB TwitterSemEval2015
          config: default
          split: test
          revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
        metrics:
          - type: cos_sim_accuracy
            value: 83.65023544137807
          - type: cos_sim_ap
            value: 65.99787692764193
          - type: cos_sim_f1
            value: 62.10650887573965
          - type: cos_sim_precision
            value: 56.30901287553648
          - type: cos_sim_recall
            value: 69.23482849604221
          - type: dot_accuracy
            value: 79.10830303391549
          - type: dot_ap
            value: 48.80109642320246
          - type: dot_f1
            value: 51.418744625967314
          - type: dot_precision
            value: 40.30253107683091
          - type: dot_recall
            value: 71.00263852242745
          - type: euclidean_accuracy
            value: 82.45812719794957
          - type: euclidean_ap
            value: 60.09969493259607
          - type: euclidean_f1
            value: 57.658573789246226
          - type: euclidean_precision
            value: 55.62913907284768
          - type: euclidean_recall
            value: 59.84168865435356
          - type: manhattan_accuracy
            value: 82.46408773916671
          - type: manhattan_ap
            value: 60.116199786815116
          - type: manhattan_f1
            value: 57.683903860160235
          - type: manhattan_precision
            value: 53.41726618705036
          - type: manhattan_recall
            value: 62.69129287598945
          - type: max_accuracy
            value: 83.65023544137807
          - type: max_ap
            value: 65.99787692764193
          - type: max_f1
            value: 62.10650887573965
      - task:
          type: PairClassification
        dataset:
          type: mteb/twitterurlcorpus-pairclassification
          name: MTEB TwitterURLCorpus
          config: default
          split: test
          revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
        metrics:
          - type: cos_sim_accuracy
            value: 88.34943920518494
          - type: cos_sim_ap
            value: 84.5428891020442
          - type: cos_sim_f1
            value: 77.09709933923172
          - type: cos_sim_precision
            value: 74.83150952967607
          - type: cos_sim_recall
            value: 79.50415768401602
          - type: dot_accuracy
            value: 84.53448208949432
          - type: dot_ap
            value: 73.96328242371995
          - type: dot_f1
            value: 70.00553786515299
          - type: dot_precision
            value: 63.58777665995976
          - type: dot_recall
            value: 77.86418232214352
          - type: euclidean_accuracy
            value: 86.87662514068381
          - type: euclidean_ap
            value: 81.45499631520235
          - type: euclidean_f1
            value: 73.46567109816063
          - type: euclidean_precision
            value: 69.71037533697381
          - type: euclidean_recall
            value: 77.6485987064983
          - type: manhattan_accuracy
            value: 86.88244654014825
          - type: manhattan_ap
            value: 81.47180273946366
          - type: manhattan_f1
            value: 73.44624393136418
          - type: manhattan_precision
            value: 70.80385852090032
          - type: manhattan_recall
            value: 76.29350169387126
          - type: max_accuracy
            value: 88.34943920518494
          - type: max_ap
            value: 84.5428891020442
          - type: max_f1
            value: 77.09709933923172

SGPT-5.8B-weightedmean-msmarco-specb-bitfit

Usage

For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt

Evaluation Results

For eval results, refer to our paper: https://arxiv.org/abs/2202.08904

Training

The model was trained with the parameters:

DataLoader:

torch.utils.data.dataloader.DataLoader of length 249592 with parameters:

{'batch_size': 2, 'sampler': 'torch.utils.data.sampler.RandomSampler', 'batch_sampler': 'torch.utils.data.sampler.BatchSampler'}

Loss:

sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss with parameters:

{'scale': 20.0, 'similarity_fct': 'cos_sim'}

Parameters of the fit()-Method:

{
    "epochs": 10,
    "evaluation_steps": 0,
    "evaluator": "NoneType",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 5e-05
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 1000,
    "weight_decay": 0.01
}

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 300, 'do_lower_case': False}) with Transformer model: GPTJModel 
  (1): Pooling({'word_embedding_dimension': 4096, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)

Citing & Authors

@article{muennighoff2022sgpt,
  title={SGPT: GPT Sentence Embeddings for Semantic Search},
  author={Muennighoff, Niklas},
  journal={arXiv preprint arXiv:2202.08904},
  year={2022}
}