|
--- |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-transformers |
|
- feature-extraction |
|
- sentence-similarity |
|
language: en |
|
license: apache-2.0 |
|
datasets: |
|
- s2orc |
|
- flax-sentence-embeddings/stackexchange_xml |
|
- ms_marco |
|
- gooaq |
|
- yahoo_answers_topics |
|
- code_search_net |
|
- search_qa |
|
- eli5 |
|
- snli |
|
- multi_nli |
|
- wikihow |
|
- natural_questions |
|
- trivia_qa |
|
- embedding-data/sentence-compression |
|
- embedding-data/flickr30k-captions |
|
- embedding-data/altlex |
|
- embedding-data/simple-wiki |
|
- embedding-data/QQP |
|
- embedding-data/SPECTER |
|
- embedding-data/PAQ_pairs |
|
- embedding-data/WikiAnswers |
|
- tum-nlp/cannot-dataset |
|
model-index: |
|
- name: all-mpnet-base-v2-negation |
|
results: |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en) |
|
config: en |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 72.6268656716418 |
|
- type: ap |
|
value: 36.40585820220466 |
|
- type: f1 |
|
value: 67.06383995428979 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_polarity |
|
name: MTEB AmazonPolarityClassification |
|
config: default |
|
split: test |
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
|
metrics: |
|
- type: accuracy |
|
value: 85.11834999999999 |
|
- type: ap |
|
value: 79.72843246428603 |
|
- type: f1 |
|
value: 85.08938287851875 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (en) |
|
config: en |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 37.788000000000004 |
|
- type: f1 |
|
value: 37.40475118737949 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-p2p |
|
name: MTEB ArxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
|
metrics: |
|
- type: v_measure |
|
value: 45.73138953773995 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-s2s |
|
name: MTEB ArxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
|
metrics: |
|
- type: v_measure |
|
value: 39.13609863309245 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/askubuntudupquestions-reranking |
|
name: MTEB AskUbuntuDupQuestions |
|
config: default |
|
split: test |
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
metrics: |
|
- type: map |
|
value: 65.56639026991134 |
|
- type: mrr |
|
value: 77.8122938926263 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/biosses-sts |
|
name: MTEB BIOSSES |
|
config: default |
|
split: test |
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 72.27098152643569 |
|
- type: cos_sim_spearman |
|
value: 71.13475338373253 |
|
- type: euclidean_pearson |
|
value: 70.48545151074218 |
|
- type: euclidean_spearman |
|
value: 69.49917394727082 |
|
- type: manhattan_pearson |
|
value: 69.2653740752147 |
|
- type: manhattan_spearman |
|
value: 68.59192435931085 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/banking77 |
|
name: MTEB Banking77Classification |
|
config: default |
|
split: test |
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
metrics: |
|
- type: accuracy |
|
value: 84.7012987012987 |
|
- type: f1 |
|
value: 84.61766470772943 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-p2p |
|
name: MTEB BiorxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
|
metrics: |
|
- type: v_measure |
|
value: 37.61314886948818 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-s2s |
|
name: MTEB BiorxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
|
metrics: |
|
- type: v_measure |
|
value: 34.496442588205205 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/emotion |
|
name: MTEB EmotionClassification |
|
config: default |
|
split: test |
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
|
metrics: |
|
- type: accuracy |
|
value: 45.63 |
|
- type: f1 |
|
value: 40.24119129248194 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/imdb |
|
name: MTEB ImdbClassification |
|
config: default |
|
split: test |
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
metrics: |
|
- type: accuracy |
|
value: 74.73479999999999 |
|
- type: ap |
|
value: 68.80435332319863 |
|
- type: f1 |
|
value: 74.66014345440416 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (en) |
|
config: en |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 93.06429548563612 |
|
- type: f1 |
|
value: 92.91686969560733 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 78.19197446420428 |
|
- type: f1 |
|
value: 61.50020940946492 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 73.86684599865502 |
|
- type: f1 |
|
value: 72.11245795864379 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (en) |
|
config: en |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 77.53866845998655 |
|
- type: f1 |
|
value: 77.51746806908895 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-p2p |
|
name: MTEB MedrxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
|
metrics: |
|
- type: v_measure |
|
value: 33.66744884855605 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-s2s |
|
name: MTEB MedrxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
|
metrics: |
|
- type: v_measure |
|
value: 31.951900966550262 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/mind_small |
|
name: MTEB MindSmallReranking |
|
config: default |
|
split: test |
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
|
metrics: |
|
- type: map |
|
value: 29.34485636178124 |
|
- type: mrr |
|
value: 30.118035109577022 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering |
|
name: MTEB RedditClustering |
|
config: default |
|
split: test |
|
revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
|
metrics: |
|
- type: v_measure |
|
value: 47.14306531904168 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering-p2p |
|
name: MTEB RedditClusteringP2P |
|
config: default |
|
split: test |
|
revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
|
metrics: |
|
- type: v_measure |
|
value: 51.59878183893005 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sickr-sts |
|
name: MTEB SICK-R |
|
config: default |
|
split: test |
|
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 78.5530506834234 |
|
- type: cos_sim_spearman |
|
value: 77.45787185404667 |
|
- type: euclidean_pearson |
|
value: 76.37727601604011 |
|
- type: euclidean_spearman |
|
value: 77.14250754925013 |
|
- type: manhattan_pearson |
|
value: 75.85855462882735 |
|
- type: manhattan_spearman |
|
value: 76.6223895689777 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts12-sts |
|
name: MTEB STS12 |
|
config: default |
|
split: test |
|
revision: a0d554a64d88156834ff5ae9920b964011b16384 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 83.1019526956277 |
|
- type: cos_sim_spearman |
|
value: 72.98362332123834 |
|
- type: euclidean_pearson |
|
value: 78.42992808997602 |
|
- type: euclidean_spearman |
|
value: 70.79569301491145 |
|
- type: manhattan_pearson |
|
value: 77.96413528436207 |
|
- type: manhattan_spearman |
|
value: 70.34707852104586 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts13-sts |
|
name: MTEB STS13 |
|
config: default |
|
split: test |
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 85.09200805966644 |
|
- type: cos_sim_spearman |
|
value: 85.52497834636847 |
|
- type: euclidean_pearson |
|
value: 84.20407512505086 |
|
- type: euclidean_spearman |
|
value: 85.35640946044332 |
|
- type: manhattan_pearson |
|
value: 83.79425758102826 |
|
- type: manhattan_spearman |
|
value: 84.9531731481683 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts14-sts |
|
name: MTEB STS14 |
|
config: default |
|
split: test |
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 82.43419245577238 |
|
- type: cos_sim_spearman |
|
value: 79.87215923164575 |
|
- type: euclidean_pearson |
|
value: 80.99628882719712 |
|
- type: euclidean_spearman |
|
value: 79.2671186335978 |
|
- type: manhattan_pearson |
|
value: 80.47076166661054 |
|
- type: manhattan_spearman |
|
value: 78.82329686631051 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts15-sts |
|
name: MTEB STS15 |
|
config: default |
|
split: test |
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.67294508915346 |
|
- type: cos_sim_spearman |
|
value: 85.34528695616378 |
|
- type: euclidean_pearson |
|
value: 83.65270617275111 |
|
- type: euclidean_spearman |
|
value: 84.64456096952591 |
|
- type: manhattan_pearson |
|
value: 83.26416114783083 |
|
- type: manhattan_spearman |
|
value: 84.26944094512996 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts16-sts |
|
name: MTEB STS16 |
|
config: default |
|
split: test |
|
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 80.70172607906416 |
|
- type: cos_sim_spearman |
|
value: 81.96031310316046 |
|
- type: euclidean_pearson |
|
value: 82.34820192315314 |
|
- type: euclidean_spearman |
|
value: 82.72576940549405 |
|
- type: manhattan_pearson |
|
value: 81.93093910116202 |
|
- type: manhattan_spearman |
|
value: 82.25431799152639 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-en) |
|
config: en-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 90.43640731744911 |
|
- type: cos_sim_spearman |
|
value: 90.16343998541602 |
|
- type: euclidean_pearson |
|
value: 89.49834342254633 |
|
- type: euclidean_spearman |
|
value: 90.17304989919288 |
|
- type: manhattan_pearson |
|
value: 89.32424382015218 |
|
- type: manhattan_spearman |
|
value: 89.91884845996768 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (en) |
|
config: en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 62.06205206393254 |
|
- type: cos_sim_spearman |
|
value: 60.920792876665885 |
|
- type: euclidean_pearson |
|
value: 60.49188637403393 |
|
- type: euclidean_spearman |
|
value: 60.73500415357452 |
|
- type: manhattan_pearson |
|
value: 59.94692152491976 |
|
- type: manhattan_spearman |
|
value: 60.215426858338994 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/stsbenchmark-sts |
|
name: MTEB STSBenchmark |
|
config: default |
|
split: test |
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.78948820087687 |
|
- type: cos_sim_spearman |
|
value: 84.64531509697663 |
|
- type: euclidean_pearson |
|
value: 84.77264321816324 |
|
- type: euclidean_spearman |
|
value: 84.67485410196043 |
|
- type: manhattan_pearson |
|
value: 84.43100272264775 |
|
- type: manhattan_spearman |
|
value: 84.29254033404217 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/scidocs-reranking |
|
name: MTEB SciDocsRR |
|
config: default |
|
split: test |
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
|
metrics: |
|
- type: map |
|
value: 88.39411601972704 |
|
- type: mrr |
|
value: 96.49192583016112 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/sprintduplicatequestions-pairclassification |
|
name: MTEB SprintDuplicateQuestions |
|
config: default |
|
split: test |
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 99.55445544554455 |
|
- type: cos_sim_ap |
|
value: 84.82462858434408 |
|
- type: cos_sim_f1 |
|
value: 76.11464968152866 |
|
- type: cos_sim_precision |
|
value: 81.10859728506787 |
|
- type: cos_sim_recall |
|
value: 71.7 |
|
- type: dot_accuracy |
|
value: 99.48613861386139 |
|
- type: dot_ap |
|
value: 80.97278220281665 |
|
- type: dot_f1 |
|
value: 72.2914669223394 |
|
- type: dot_precision |
|
value: 69.42909760589319 |
|
- type: dot_recall |
|
value: 75.4 |
|
- type: euclidean_accuracy |
|
value: 99.56138613861386 |
|
- type: euclidean_ap |
|
value: 85.21566333946467 |
|
- type: euclidean_f1 |
|
value: 76.60239708181345 |
|
- type: euclidean_precision |
|
value: 79.97823721436343 |
|
- type: euclidean_recall |
|
value: 73.5 |
|
- type: manhattan_accuracy |
|
value: 99.55148514851486 |
|
- type: manhattan_ap |
|
value: 84.49960192851891 |
|
- type: manhattan_f1 |
|
value: 75.9681697612732 |
|
- type: manhattan_precision |
|
value: 80.90395480225989 |
|
- type: manhattan_recall |
|
value: 71.6 |
|
- type: max_accuracy |
|
value: 99.56138613861386 |
|
- type: max_ap |
|
value: 85.21566333946467 |
|
- type: max_f1 |
|
value: 76.60239708181345 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering |
|
name: MTEB StackExchangeClustering |
|
config: default |
|
split: test |
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
|
metrics: |
|
- type: v_measure |
|
value: 49.33929838947165 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering-p2p |
|
name: MTEB StackExchangeClusteringP2P |
|
config: default |
|
split: test |
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
|
metrics: |
|
- type: v_measure |
|
value: 31.523973661953686 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/stackoverflowdupquestions-reranking |
|
name: MTEB StackOverflowDupQuestions |
|
config: default |
|
split: test |
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
|
metrics: |
|
- type: map |
|
value: 52.22408767861519 |
|
- type: mrr |
|
value: 53.16279921059333 |
|
- task: |
|
type: Summarization |
|
dataset: |
|
type: mteb/summeval |
|
name: MTEB SummEval |
|
config: default |
|
split: test |
|
revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 28.128173244098726 |
|
- type: cos_sim_spearman |
|
value: 30.149225143523662 |
|
- type: dot_pearson |
|
value: 24.322914168643386 |
|
- type: dot_spearman |
|
value: 26.38194545372431 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/toxic_conversations_50k |
|
name: MTEB ToxicConversationsClassification |
|
config: default |
|
split: test |
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
|
metrics: |
|
- type: accuracy |
|
value: 67.6684 |
|
- type: ap |
|
value: 12.681984793717413 |
|
- type: f1 |
|
value: 51.97637585601529 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/tweet_sentiment_extraction |
|
name: MTEB TweetSentimentExtractionClassification |
|
config: default |
|
split: test |
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
metrics: |
|
- type: accuracy |
|
value: 58.44086021505377 |
|
- type: f1 |
|
value: 58.68058329615692 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/twentynewsgroups-clustering |
|
name: MTEB TwentyNewsgroupsClustering |
|
config: default |
|
split: test |
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
|
metrics: |
|
- type: v_measure |
|
value: 44.226944341054015 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twittersemeval2015-pairclassification |
|
name: MTEB TwitterSemEval2015 |
|
config: default |
|
split: test |
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 86.87488823985218 |
|
- type: cos_sim_ap |
|
value: 76.85283892335002 |
|
- type: cos_sim_f1 |
|
value: 70.42042042042041 |
|
- type: cos_sim_precision |
|
value: 66.96811042360781 |
|
- type: cos_sim_recall |
|
value: 74.24802110817942 |
|
- type: dot_accuracy |
|
value: 84.85426476724086 |
|
- type: dot_ap |
|
value: 70.77036812650887 |
|
- type: dot_f1 |
|
value: 66.4901577069184 |
|
- type: dot_precision |
|
value: 58.97488258117215 |
|
- type: dot_recall |
|
value: 76.2005277044855 |
|
- type: euclidean_accuracy |
|
value: 86.95833581689217 |
|
- type: euclidean_ap |
|
value: 77.05903224969623 |
|
- type: euclidean_f1 |
|
value: 70.75323419175432 |
|
- type: euclidean_precision |
|
value: 65.2979245704084 |
|
- type: euclidean_recall |
|
value: 77.20316622691293 |
|
- type: manhattan_accuracy |
|
value: 86.88084878106932 |
|
- type: manhattan_ap |
|
value: 76.95056209047733 |
|
- type: manhattan_f1 |
|
value: 70.61542203843348 |
|
- type: manhattan_precision |
|
value: 65.50090252707581 |
|
- type: manhattan_recall |
|
value: 76.59630606860158 |
|
- type: max_accuracy |
|
value: 86.95833581689217 |
|
- type: max_ap |
|
value: 77.05903224969623 |
|
- type: max_f1 |
|
value: 70.75323419175432 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twitterurlcorpus-pairclassification |
|
name: MTEB TwitterURLCorpus |
|
config: default |
|
split: test |
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 88.43870066363954 |
|
- type: cos_sim_ap |
|
value: 84.77197321507954 |
|
- type: cos_sim_f1 |
|
value: 76.91440595175472 |
|
- type: cos_sim_precision |
|
value: 75.11375311903713 |
|
- type: cos_sim_recall |
|
value: 78.80351093316908 |
|
- type: dot_accuracy |
|
value: 87.60624054022587 |
|
- type: dot_ap |
|
value: 83.16574114504616 |
|
- type: dot_f1 |
|
value: 75.5050226294293 |
|
- type: dot_precision |
|
value: 72.30953555571217 |
|
- type: dot_recall |
|
value: 78.99599630428088 |
|
- type: euclidean_accuracy |
|
value: 88.2951061435169 |
|
- type: euclidean_ap |
|
value: 84.28559058741602 |
|
- type: euclidean_f1 |
|
value: 76.7921146953405 |
|
- type: euclidean_precision |
|
value: 74.54334589736156 |
|
- type: euclidean_recall |
|
value: 79.1807822605482 |
|
- type: manhattan_accuracy |
|
value: 88.23883261536074 |
|
- type: manhattan_ap |
|
value: 84.20593815258039 |
|
- type: manhattan_f1 |
|
value: 76.74366281685916 |
|
- type: manhattan_precision |
|
value: 74.80263157894737 |
|
- type: manhattan_recall |
|
value: 78.78811210348013 |
|
- type: max_accuracy |
|
value: 88.43870066363954 |
|
- type: max_ap |
|
value: 84.77197321507954 |
|
- type: max_f1 |
|
value: 76.91440595175472 |
|
--- |
|
|
|
# all-mpnet-base-v2-negation |
|
**This is a fine-tuned [sentence-transformers](https://www.SBERT.net) model to perform better on negated pairs of sentences.** |
|
|
|
It maps sentences and paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search. |
|
|
|
## Usage (Sentence-Transformers) |
|
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: |
|
|
|
``` |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
Then you can use the model like this: |
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
|
|
sentences = [ |
|
"I like rainy days because they make me feel relaxed.", |
|
"I don't like rainy days because they don't make me feel relaxed." |
|
] |
|
|
|
model = SentenceTransformer('dmlls/all-mpnet-base-v2-negation') |
|
embeddings = model.encode(sentences) |
|
print(embeddings) |
|
``` |
|
|
|
## Usage (HuggingFace Transformers) |
|
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings. |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModel |
|
import torch |
|
import torch.nn.functional as F |
|
|
|
# Mean Pooling - Take attention mask into account for correct averaging |
|
def mean_pooling(model_output, attention_mask): |
|
token_embeddings = model_output[0] #First element of model_output contains all token embeddings |
|
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float() |
|
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9) |
|
|
|
|
|
# Sentences we want sentence embeddings for |
|
sentences = [ |
|
"I like rainy days because they make me feel relaxed.", |
|
"I don't like rainy days because they don't make me feel relaxed." |
|
] |
|
|
|
# Load model from HuggingFace Hub |
|
tokenizer = AutoTokenizer.from_pretrained('dmlls/all-mpnet-base-v2-negation') |
|
model = AutoModel.from_pretrained('dmlls/all-mpnet-base-v2-negation') |
|
|
|
# Tokenize sentences |
|
encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt') |
|
|
|
# Compute token embeddings |
|
with torch.no_grad(): |
|
model_output = model(**encoded_input) |
|
|
|
# Perform pooling |
|
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask']) |
|
|
|
# Normalize embeddings |
|
sentence_embeddings = F.normalize(sentence_embeddings, p=2, dim=1) |
|
|
|
print(sentence_embeddings) |
|
``` |
|
|
|
------ |
|
|
|
## Background |
|
|
|
This model was finetuned within the context of the [*This is not correct! Negation-aware Evaluation of Language Generation Systems*](https://arxiv.org/abs/2307.13989) paper. |
|
|
|
|
|
## Intended uses |
|
|
|
Our model is intended to be used as a sentence and short paragraph encoder, performing well (i.e., reporting lower similarity scores) on negated pairs of sentences when compared to its base model. |
|
|
|
Given an input text, it outputs a vector which captures the semantic information. The sentence vector may be used for information retrieval, clustering or sentence similarity tasks. |
|
|
|
By default, input text longer than 384 word pieces is truncated. |
|
|
|
|
|
## Training procedure |
|
|
|
### Pre-training |
|
|
|
We used [`sentence-transformers/all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) as base model. |
|
|
|
### Fine-tuning |
|
|
|
We fine-tuned the model on the [CANNOT dataset](https://huggingface.co/datasets/tum-nlp/cannot-dataset) using a contrastive objective. Formally, we compute the cosine similarity from each possible sentence pairs from the batch. We then apply the cross entropy loss by comparing with true pairs. |
|
|
|
#### Hyper parameters |
|
We followed an analogous approach to [how other Sentence Transformers were trained](https://github.com/UKPLab/sentence-transformers/blob/3e1929fddef16df94f8bc6e3b10598a98f46e62d/examples/training/nli/training_nli_v2.py). We took the first 90% of samples from the CANNOT dataset as the training split. |
|
We used a batch size of 64 and trained for 1 epoch. |
