Hugging Face's logo

dunzhang
/
stella-mrl-large-zh-v3.5-1792d

Sentence Similarity
sentence-transformers
PyTorch
bert
feature-extraction
mteb
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
12
infgrad commited on
Commit
de06106
•
1 Parent(s): c4bc1fd

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1115 -1
README.md CHANGED
@@ -1,3 +1,1117 @@
1
  ---
2
- license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ pipeline_tag: sentence-similarity
3
+ tags:
4
+ - sentence-transformers
5
+ - feature-extraction
6
+ - sentence-similarity
7
+ - mteb
8
+ model-index:
9
+ - name: stella-mrl-large-zh-v3.5-1792d
10
+ results:
11
+ - task:
12
+ type: STS
13
+ dataset:
14
+ type: C-MTEB/AFQMC
15
+ name: MTEB AFQMC
16
+ config: default
17
+ split: validation
18
+ revision: None
19
+ metrics:
20
+ - type: cos_sim_pearson
21
+ value: 54.33822814973567
22
+ - type: cos_sim_spearman
23
+ value: 58.85457316132848
24
+ - type: euclidean_pearson
25
+ value: 57.57048145477383
26
+ - type: euclidean_spearman
27
+ value: 58.854593263425095
28
+ - type: manhattan_pearson
29
+ value: 57.55884028558309
30
+ - type: manhattan_spearman
31
+ value: 58.84474216217465
32
+ - task:
33
+ type: STS
34
+ dataset:
35
+ type: C-MTEB/ATEC
36
+ name: MTEB ATEC
37
+ config: default
38
+ split: test
39
+ revision: None
40
+ metrics:
41
+ - type: cos_sim_pearson
42
+ value: 54.219652875381875
43
+ - type: cos_sim_spearman
44
+ value: 58.079506691583546
45
+ - type: euclidean_pearson
46
+ value: 61.646366330471736
47
+ - type: euclidean_spearman
48
+ value: 58.07951006894859
49
+ - type: manhattan_pearson
50
+ value: 61.64460832085762
51
+ - type: manhattan_spearman
52
+ value: 58.08054699349972
53
+ - task:
54
+ type: Classification
55
+ dataset:
56
+ type: mteb/amazon_reviews_multi
57
+ name: MTEB AmazonReviewsClassification (zh)
58
+ config: zh
59
+ split: test
60
+ revision: 1399c76144fd37290681b995c656ef9b2e06e26d
61
+ metrics:
62
+ - type: accuracy
63
+ value: 46.593999999999994
64
+ - type: f1
65
+ value: 44.73150848183217
66
+ - task:
67
+ type: STS
68
+ dataset:
69
+ type: C-MTEB/BQ
70
+ name: MTEB BQ
71
+ config: default
72
+ split: test
73
+ revision: None
74
+ metrics:
75
+ - type: cos_sim_pearson
76
+ value: 69.16841007040091
77
+ - type: cos_sim_spearman
78
+ value: 71.04760904227217
79
+ - type: euclidean_pearson
80
+ value: 69.95126084376611
81
+ - type: euclidean_spearman
82
+ value: 71.04760904184589
83
+ - type: manhattan_pearson
84
+ value: 69.92512024129407
85
+ - type: manhattan_spearman
86
+ value: 71.02613161257672
87
+ - task:
88
+ type: Clustering
89
+ dataset:
90
+ type: C-MTEB/CLSClusteringP2P
91
+ name: MTEB CLSClusteringP2P
92
+ config: default
93
+ split: test
94
+ revision: None
95
+ metrics:
96
+ - type: v_measure
97
+ value: 43.032332399653306
98
+ - task:
99
+ type: Clustering
100
+ dataset:
101
+ type: C-MTEB/CLSClusteringS2S
102
+ name: MTEB CLSClusteringS2S
103
+ config: default
104
+ split: test
105
+ revision: None
106
+ metrics:
107
+ - type: v_measure
108
+ value: 40.41603958793544
109
+ - task:
110
+ type: Reranking
111
+ dataset:
112
+ type: C-MTEB/CMedQAv1-reranking
113
+ name: MTEB CMedQAv1
114
+ config: default
115
+ split: test
116
+ revision: None
117
+ metrics:
118
+ - type: map
119
+ value: 89.33487924447584
120
+ - type: mrr
121
+ value: 91.34623015873017
122
+ - task:
123
+ type: Reranking
124
+ dataset:
125
+ type: C-MTEB/CMedQAv2-reranking
126
+ name: MTEB CMedQAv2
127
+ config: default
128
+ split: test
129
+ revision: None
130
+ metrics:
131
+ - type: map
132
+ value: 89.17795270698021
133
+ - type: mrr
134
+ value: 91.0956746031746
135
+ - task:
136
+ type: Retrieval
137
+ dataset:
138
+ type: C-MTEB/CmedqaRetrieval
139
+ name: MTEB CmedqaRetrieval
140
+ config: default
141
+ split: dev
142
+ revision: None
143
+ metrics:
144
+ - type: map_at_1
145
+ value: 26.809
146
+ - type: map_at_10
147
+ value: 39.906000000000006
148
+ - type: map_at_100
149
+ value: 41.858000000000004
150
+ - type: map_at_1000
151
+ value: 41.954
152
+ - type: map_at_3
153
+ value: 35.435
154
+ - type: map_at_5
155
+ value: 37.978
156
+ - type: mrr_at_1
157
+ value: 40.660000000000004
158
+ - type: mrr_at_10
159
+ value: 48.787000000000006
160
+ - type: mrr_at_100
161
+ value: 49.796
162
+ - type: mrr_at_1000
163
+ value: 49.832
164
+ - type: mrr_at_3
165
+ value: 46.166000000000004
166
+ - type: mrr_at_5
167
+ value: 47.675
168
+ - type: ndcg_at_1
169
+ value: 40.660000000000004
170
+ - type: ndcg_at_10
171
+ value: 46.614
172
+ - type: ndcg_at_100
173
+ value: 54.037
174
+ - type: ndcg_at_1000
175
+ value: 55.654
176
+ - type: ndcg_at_3
177
+ value: 41.032000000000004
178
+ - type: ndcg_at_5
179
+ value: 43.464999999999996
180
+ - type: precision_at_1
181
+ value: 40.660000000000004
182
+ - type: precision_at_10
183
+ value: 10.35
184
+ - type: precision_at_100
185
+ value: 1.6340000000000001
186
+ - type: precision_at_1000
187
+ value: 0.184
188
+ - type: precision_at_3
189
+ value: 23.122
190
+ - type: precision_at_5
191
+ value: 16.944
192
+ - type: recall_at_1
193
+ value: 26.809
194
+ - type: recall_at_10
195
+ value: 57.474000000000004
196
+ - type: recall_at_100
197
+ value: 87.976
198
+ - type: recall_at_1000
199
+ value: 98.74199999999999
200
+ - type: recall_at_3
201
+ value: 40.819
202
+ - type: recall_at_5
203
+ value: 48.175000000000004
204
+ - task:
205
+ type: PairClassification
206
+ dataset:
207
+ type: C-MTEB/CMNLI
208
+ name: MTEB Cmnli
209
+ config: default
210
+ split: validation
211
+ revision: None
212
+ metrics:
213
+ - type: cos_sim_accuracy
214
+ value: 83.4996993385448
215
+ - type: cos_sim_ap
216
+ value: 90.66238348446467
217
+ - type: cos_sim_f1
218
+ value: 84.39077936333699
219
+ - type: cos_sim_precision
220
+ value: 79.53651975998345
221
+ - type: cos_sim_recall
222
+ value: 89.87608136544307
223
+ - type: dot_accuracy
224
+ value: 83.4996993385448
225
+ - type: dot_ap
226
+ value: 90.64660919236363
227
+ - type: dot_f1
228
+ value: 84.39077936333699
229
+ - type: dot_precision
230
+ value: 79.53651975998345
231
+ - type: dot_recall
232
+ value: 89.87608136544307
233
+ - type: euclidean_accuracy
234
+ value: 83.4996993385448
235
+ - type: euclidean_ap
236
+ value: 90.66238269557765
237
+ - type: euclidean_f1
238
+ value: 84.39077936333699
239
+ - type: euclidean_precision
240
+ value: 79.53651975998345
241
+ - type: euclidean_recall
242
+ value: 89.87608136544307
243
+ - type: manhattan_accuracy
244
+ value: 83.35538184004811
245
+ - type: manhattan_ap
246
+ value: 90.6446013420276
247
+ - type: manhattan_f1
248
+ value: 84.37465196569775
249
+ - type: manhattan_precision
250
+ value: 80.5614632071459
251
+ - type: manhattan_recall
252
+ value: 88.56675239653963
253
+ - type: max_accuracy
254
+ value: 83.4996993385448
255
+ - type: max_ap
256
+ value: 90.66238348446467
257
+ - type: max_f1
258
+ value: 84.39077936333699
259
+ - task:
260
+ type: Retrieval
261
+ dataset:
262
+ type: C-MTEB/CovidRetrieval
263
+ name: MTEB CovidRetrieval
264
+ config: default
265
+ split: dev
266
+ revision: None
267
+ metrics:
268
+ - type: map_at_1
269
+ value: 68.967
270
+ - type: map_at_10
271
+ value: 77.95299999999999
272
+ - type: map_at_100
273
+ value: 78.213
274
+ - type: map_at_1000
275
+ value: 78.21900000000001
276
+ - type: map_at_3
277
+ value: 76.30799999999999
278
+ - type: map_at_5
279
+ value: 77.316
280
+ - type: mrr_at_1
281
+ value: 69.125
282
+ - type: mrr_at_10
283
+ value: 77.886
284
+ - type: mrr_at_100
285
+ value: 78.141
286
+ - type: mrr_at_1000
287
+ value: 78.147
288
+ - type: mrr_at_3
289
+ value: 76.291
290
+ - type: mrr_at_5
291
+ value: 77.29700000000001
292
+ - type: ndcg_at_1
293
+ value: 69.231
294
+ - type: ndcg_at_10
295
+ value: 81.867
296
+ - type: ndcg_at_100
297
+ value: 82.982
298
+ - type: ndcg_at_1000
299
+ value: 83.12
300
+ - type: ndcg_at_3
301
+ value: 78.592
302
+ - type: ndcg_at_5
303
+ value: 80.39
304
+ - type: precision_at_1
305
+ value: 69.231
306
+ - type: precision_at_10
307
+ value: 9.494
308
+ - type: precision_at_100
309
+ value: 0.9990000000000001
310
+ - type: precision_at_1000
311
+ value: 0.101
312
+ - type: precision_at_3
313
+ value: 28.591
314
+ - type: precision_at_5
315
+ value: 18.061
316
+ - type: recall_at_1
317
+ value: 68.967
318
+ - type: recall_at_10
319
+ value: 93.941
320
+ - type: recall_at_100
321
+ value: 98.84100000000001
322
+ - type: recall_at_1000
323
+ value: 99.895
324
+ - type: recall_at_3
325
+ value: 85.142
326
+ - type: recall_at_5
327
+ value: 89.46300000000001
328
+ - task:
329
+ type: Retrieval
330
+ dataset:
331
+ type: C-MTEB/DuRetrieval
332
+ name: MTEB DuRetrieval
333
+ config: default
334
+ split: dev
335
+ revision: None
336
+ metrics:
337
+ - type: map_at_1
338
+ value: 25.824
339
+ - type: map_at_10
340
+ value: 79.396
341
+ - type: map_at_100
342
+ value: 82.253
343
+ - type: map_at_1000
344
+ value: 82.295
345
+ - type: map_at_3
346
+ value: 54.83
347
+ - type: map_at_5
348
+ value: 69.536
349
+ - type: mrr_at_1
350
+ value: 89.7
351
+ - type: mrr_at_10
352
+ value: 92.929
353
+ - type: mrr_at_100
354
+ value: 93.013
355
+ - type: mrr_at_1000
356
+ value: 93.015
357
+ - type: mrr_at_3
358
+ value: 92.658
359
+ - type: mrr_at_5
360
+ value: 92.841
361
+ - type: ndcg_at_1
362
+ value: 89.7
363
+ - type: ndcg_at_10
364
+ value: 86.797
365
+ - type: ndcg_at_100
366
+ value: 89.652
367
+ - type: ndcg_at_1000
368
+ value: 90.047
369
+ - type: ndcg_at_3
370
+ value: 85.651
371
+ - type: ndcg_at_5
372
+ value: 84.747
373
+ - type: precision_at_1
374
+ value: 89.7
375
+ - type: precision_at_10
376
+ value: 41.61
377
+ - type: precision_at_100
378
+ value: 4.788
379
+ - type: precision_at_1000
380
+ value: 0.488
381
+ - type: precision_at_3
382
+ value: 76.833
383
+ - type: precision_at_5
384
+ value: 65.14
385
+ - type: recall_at_1
386
+ value: 25.824
387
+ - type: recall_at_10
388
+ value: 87.896
389
+ - type: recall_at_100
390
+ value: 97.221
391
+ - type: recall_at_1000
392
+ value: 99.29599999999999
393
+ - type: recall_at_3
394
+ value: 57.178
395
+ - type: recall_at_5
396
+ value: 74.348
397
+ - task:
398
+ type: Retrieval
399
+ dataset:
400
+ type: C-MTEB/EcomRetrieval
401
+ name: MTEB EcomRetrieval
402
+ config: default
403
+ split: dev
404
+ revision: None
405
+ metrics:
406
+ - type: map_at_1
407
+ value: 52.5
408
+ - type: map_at_10
409
+ value: 63.04
410
+ - type: map_at_100
411
+ value: 63.548
412
+ - type: map_at_1000
413
+ value: 63.56
414
+ - type: map_at_3
415
+ value: 60.483
416
+ - type: map_at_5
417
+ value: 62.22800000000001
418
+ - type: mrr_at_1
419
+ value: 52.5
420
+ - type: mrr_at_10
421
+ value: 63.04
422
+ - type: mrr_at_100
423
+ value: 63.548
424
+ - type: mrr_at_1000
425
+ value: 63.56
426
+ - type: mrr_at_3
427
+ value: 60.483
428
+ - type: mrr_at_5
429
+ value: 62.22800000000001
430
+ - type: ndcg_at_1
431
+ value: 52.5
432
+ - type: ndcg_at_10
433
+ value: 68.099
434
+ - type: ndcg_at_100
435
+ value: 70.48400000000001
436
+ - type: ndcg_at_1000
437
+ value: 70.769
438
+ - type: ndcg_at_3
439
+ value: 63.01
440
+ - type: ndcg_at_5
441
+ value: 66.148
442
+ - type: precision_at_1
443
+ value: 52.5
444
+ - type: precision_at_10
445
+ value: 8.39
446
+ - type: precision_at_100
447
+ value: 0.9490000000000001
448
+ - type: precision_at_1000
449
+ value: 0.097
450
+ - type: precision_at_3
451
+ value: 23.433
452
+ - type: precision_at_5
453
+ value: 15.58
454
+ - type: recall_at_1
455
+ value: 52.5
456
+ - type: recall_at_10
457
+ value: 83.89999999999999
458
+ - type: recall_at_100
459
+ value: 94.89999999999999
460
+ - type: recall_at_1000
461
+ value: 97.1
462
+ - type: recall_at_3
463
+ value: 70.3
464
+ - type: recall_at_5
465
+ value: 77.9
466
+ - task:
467
+ type: Classification
468
+ dataset:
469
+ type: C-MTEB/IFlyTek-classification
470
+ name: MTEB IFlyTek
471
+ config: default
472
+ split: validation
473
+ revision: None
474
+ metrics:
475
+ - type: accuracy
476
+ value: 50.742593305117346
477
+ - type: f1
478
+ value: 38.7451988564002
479
+ - task:
480
+ type: Classification
481
+ dataset:
482
+ type: C-MTEB/JDReview-classification
483
+ name: MTEB JDReview
484
+ config: default
485
+ split: test
486
+ revision: None
487
+ metrics:
488
+ - type: accuracy
489
+ value: 86.09756097560977
490
+ - type: ap
491
+ value: 54.39255221143281
492
+ - type: f1
493
+ value: 80.8326851537251
494
+ - task:
495
+ type: STS
496
+ dataset:
497
+ type: C-MTEB/LCQMC
498
+ name: MTEB LCQMC
499
+ config: default
500
+ split: test
501
+ revision: None
502
+ metrics:
503
+ - type: cos_sim_pearson
504
+ value: 72.32408066246728
505
+ - type: cos_sim_spearman
506
+ value: 78.25773378380241
507
+ - type: euclidean_pearson
508
+ value: 77.87824677060661
509
+ - type: euclidean_spearman
510
+ value: 78.25773599854358
511
+ - type: manhattan_pearson
512
+ value: 77.86648277798515
513
+ - type: manhattan_spearman
514
+ value: 78.24642917155661
515
+ - task:
516
+ type: Reranking
517
+ dataset:
518
+ type: C-MTEB/Mmarco-reranking
519
+ name: MTEB MMarcoReranking
520
+ config: default
521
+ split: dev
522
+ revision: None
523
+ metrics:
524
+ - type: map
525
+ value: 28.846601097874608
526
+ - type: mrr
527
+ value: 27.902777777777775
528
+ - task:
529
+ type: Retrieval
530
+ dataset:
531
+ type: C-MTEB/MMarcoRetrieval
532
+ name: MTEB MMarcoRetrieval
533
+ config: default
534
+ split: dev
535
+ revision: None
536
+ metrics:
537
+ - type: map_at_1
538
+ value: 66.533
539
+ - type: map_at_10
540
+ value: 75.58399999999999
541
+ - type: map_at_100
542
+ value: 75.91
543
+ - type: map_at_1000
544
+ value: 75.921
545
+ - type: map_at_3
546
+ value: 73.847
547
+ - type: map_at_5
548
+ value: 74.929
549
+ - type: mrr_at_1
550
+ value: 68.854
551
+ - type: mrr_at_10
552
+ value: 76.20700000000001
553
+ - type: mrr_at_100
554
+ value: 76.498
555
+ - type: mrr_at_1000
556
+ value: 76.508
557
+ - type: mrr_at_3
558
+ value: 74.71600000000001
559
+ - type: mrr_at_5
560
+ value: 75.653
561
+ - type: ndcg_at_1
562
+ value: 68.854
563
+ - type: ndcg_at_10
564
+ value: 79.209
565
+ - type: ndcg_at_100
566
+ value: 80.67
567
+ - type: ndcg_at_1000
568
+ value: 80.95
569
+ - type: ndcg_at_3
570
+ value: 75.923
571
+ - type: ndcg_at_5
572
+ value: 77.74799999999999
573
+ - type: precision_at_1
574
+ value: 68.854
575
+ - type: precision_at_10
576
+ value: 9.547
577
+ - type: precision_at_100
578
+ value: 1.027
579
+ - type: precision_at_1000
580
+ value: 0.105
581
+ - type: precision_at_3
582
+ value: 28.582
583
+ - type: precision_at_5
584
+ value: 18.112000000000002
585
+ - type: recall_at_1
586
+ value: 66.533
587
+ - type: recall_at_10
588
+ value: 89.736
589
+ - type: recall_at_100
590
+ value: 96.34
591
+ - type: recall_at_1000
592
+ value: 98.52
593
+ - type: recall_at_3
594
+ value: 81.047
595
+ - type: recall_at_5
596
+ value: 85.38900000000001
597
+ - task:
598
+ type: Classification
599
+ dataset:
600
+ type: mteb/amazon_massive_intent
601
+ name: MTEB MassiveIntentClassification (zh-CN)
602
+ config: zh-CN
603
+ split: test
604
+ revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
605
+ metrics:
606
+ - type: accuracy
607
+ value: 73.27841291190316
608
+ - type: f1
609
+ value: 70.82287701665152
610
+ - task:
611
+ type: Classification
612
+ dataset:
613
+ type: mteb/amazon_massive_scenario
614
+ name: MTEB MassiveScenarioClassification (zh-CN)
615
+ config: zh-CN
616
+ split: test
617
+ revision: 7d571f92784cd94a019292a1f45445077d0ef634
618
+ metrics:
619
+ - type: accuracy
620
+ value: 76.20040349697376
621
+ - type: f1
622
+ value: 75.92782428878164
623
+ - task:
624
+ type: Retrieval
625
+ dataset:
626
+ type: C-MTEB/MedicalRetrieval
627
+ name: MTEB MedicalRetrieval
628
+ config: default
629
+ split: dev
630
+ revision: None
631
+ metrics:
632
+ - type: map_at_1
633
+ value: 56.39999999999999
634
+ - type: map_at_10
635
+ value: 62.122
636
+ - type: map_at_100
637
+ value: 62.692
638
+ - type: map_at_1000
639
+ value: 62.739
640
+ - type: map_at_3
641
+ value: 60.617
642
+ - type: map_at_5
643
+ value: 61.582
644
+ - type: mrr_at_1
645
+ value: 56.39999999999999
646
+ - type: mrr_at_10
647
+ value: 62.125
648
+ - type: mrr_at_100
649
+ value: 62.696
650
+ - type: mrr_at_1000
651
+ value: 62.742
652
+ - type: mrr_at_3
653
+ value: 60.617
654
+ - type: mrr_at_5
655
+ value: 61.602000000000004
656
+ - type: ndcg_at_1
657
+ value: 56.39999999999999
658
+ - type: ndcg_at_10
659
+ value: 64.986
660
+ - type: ndcg_at_100
661
+ value: 67.889
662
+ - type: ndcg_at_1000
663
+ value: 69.16499999999999
664
+ - type: ndcg_at_3
665
+ value: 61.951
666
+ - type: ndcg_at_5
667
+ value: 63.685
668
+ - type: precision_at_1
669
+ value: 56.39999999999999
670
+ - type: precision_at_10
671
+ value: 7.3999999999999995
672
+ - type: precision_at_100
673
+ value: 0.8789999999999999
674
+ - type: precision_at_1000
675
+ value: 0.098
676
+ - type: precision_at_3
677
+ value: 21.933
678
+ - type: precision_at_5
679
+ value: 14.000000000000002
680
+ - type: recall_at_1
681
+ value: 56.39999999999999
682
+ - type: recall_at_10
683
+ value: 74.0
684
+ - type: recall_at_100
685
+ value: 87.9
686
+ - type: recall_at_1000
687
+ value: 98.0
688
+ - type: recall_at_3
689
+ value: 65.8
690
+ - type: recall_at_5
691
+ value: 70.0
692
+ - task:
693
+ type: Classification
694
+ dataset:
695
+ type: C-MTEB/MultilingualSentiment-classification
696
+ name: MTEB MultilingualSentiment
697
+ config: default
698
+ split: validation
699
+ revision: None
700
+ metrics:
701
+ - type: accuracy
702
+ value: 76.64
703
+ - type: f1
704
+ value: 76.5446299028248
705
+ - task:
706
+ type: PairClassification
707
+ dataset:
708
+ type: C-MTEB/OCNLI
709
+ name: MTEB Ocnli
710
+ config: default
711
+ split: validation
712
+ revision: None
713
+ metrics:
714
+ - type: cos_sim_accuracy
715
+ value: 82.34975636166757
716
+ - type: cos_sim_ap
717
+ value: 85.51352392694149
718
+ - type: cos_sim_f1
719
+ value: 83.53057199211045
720
+ - type: cos_sim_precision
721
+ value: 78.35337650323775
722
+ - type: cos_sim_recall
723
+ value: 89.44033790918691
724
+ - type: dot_accuracy
725
+ value: 82.34975636166757
726
+ - type: dot_ap
727
+ value: 85.51347115601486
728
+ - type: dot_f1
729
+ value: 83.53057199211045
730
+ - type: dot_precision
731
+ value: 78.35337650323775
732
+ - type: dot_recall
733
+ value: 89.44033790918691
734
+ - type: euclidean_accuracy
735
+ value: 82.34975636166757
736
+ - type: euclidean_ap
737
+ value: 85.51352392694149
738
+ - type: euclidean_f1
739
+ value: 83.53057199211045
740
+ - type: euclidean_precision
741
+ value: 78.35337650323775
742
+ - type: euclidean_recall
743
+ value: 89.44033790918691
744
+ - type: manhattan_accuracy
745
+ value: 82.34975636166757
746
+ - type: manhattan_ap
747
+ value: 85.48313896880585
748
+ - type: manhattan_f1
749
+ value: 83.52414136386261
750
+ - type: manhattan_precision
751
+ value: 79.00188323917138
752
+ - type: manhattan_recall
753
+ value: 88.59556494192185
754
+ - type: max_accuracy
755
+ value: 82.34975636166757
756
+ - type: max_ap
757
+ value: 85.51352392694149
758
+ - type: max_f1
759
+ value: 83.53057199211045
760
+ - task:
761
+ type: Classification
762
+ dataset:
763
+ type: C-MTEB/OnlineShopping-classification
764
+ name: MTEB OnlineShopping
765
+ config: default
766
+ split: test
767
+ revision: None
768
+ metrics:
769
+ - type: accuracy
770
+ value: 93.39
771
+ - type: ap
772
+ value: 91.62127505252761
773
+ - type: f1
774
+ value: 93.38126146765326
775
+ - task:
776
+ type: STS
777
+ dataset:
778
+ type: C-MTEB/PAWSX
779
+ name: MTEB PAWSX
780
+ config: default
781
+ split: test
782
+ revision: None
783
+ metrics:
784
+ - type: cos_sim_pearson
785
+ value: 39.69424895486595
786
+ - type: cos_sim_spearman
787
+ value: 45.357868735202885
788
+ - type: euclidean_pearson
789
+ value: 44.85027304963503
790
+ - type: euclidean_spearman
791
+ value: 45.356945176162064
792
+ - type: manhattan_pearson
793
+ value: 44.866080721344744
794
+ - type: manhattan_spearman
795
+ value: 45.37053172312661
796
+ - task:
797
+ type: STS
798
+ dataset:
799
+ type: C-MTEB/QBQTC
800
+ name: MTEB QBQTC
801
+ config: default
802
+ split: test
803
+ revision: None
804
+ metrics:
805
+ - type: cos_sim_pearson
806
+ value: 37.03908089465844
807
+ - type: cos_sim_spearman
808
+ value: 38.98314179826781
809
+ - type: euclidean_pearson
810
+ value: 37.189386019789545
811
+ - type: euclidean_spearman
812
+ value: 38.98311189555396
813
+ - type: manhattan_pearson
814
+ value: 37.14695118899785
815
+ - type: manhattan_spearman
816
+ value: 38.94957261261034
817
+ - task:
818
+ type: STS
819
+ dataset:
820
+ type: mteb/sts22-crosslingual-sts
821
+ name: MTEB STS22 (zh)
822
+ config: zh
823
+ split: test
824
+ revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
825
+ metrics:
826
+ - type: cos_sim_pearson
827
+ value: 65.08396305098712
828
+ - type: cos_sim_spearman
829
+ value: 66.26346934994216
830
+ - type: euclidean_pearson
831
+ value: 65.56501615370941
832
+ - type: euclidean_spearman
833
+ value: 66.26346934994216
834
+ - type: manhattan_pearson
835
+ value: 65.47984748172154
836
+ - type: manhattan_spearman
837
+ value: 66.25326746119808
838
+ - task:
839
+ type: STS
840
+ dataset:
841
+ type: C-MTEB/STSB
842
+ name: MTEB STSB
843
+ config: default
844
+ split: test
845
+ revision: None
846
+ metrics:
847
+ - type: cos_sim_pearson
848
+ value: 80.95965207330296
849
+ - type: cos_sim_spearman
850
+ value: 82.96149593569953
851
+ - type: euclidean_pearson
852
+ value: 82.67125448003975
853
+ - type: euclidean_spearman
854
+ value: 82.96141174550262
855
+ - type: manhattan_pearson
856
+ value: 82.64660468206361
857
+ - type: manhattan_spearman
858
+ value: 82.91756025324656
859
+ - task:
860
+ type: Reranking
861
+ dataset:
862
+ type: C-MTEB/T2Reranking
863
+ name: MTEB T2Reranking
864
+ config: default
865
+ split: dev
866
+ revision: None
867
+ metrics:
868
+ - type: map
869
+ value: 66.43391960680063
870
+ - type: mrr
871
+ value: 76.078440855015
872
+ - task:
873
+ type: Retrieval
874
+ dataset:
875
+ type: C-MTEB/T2Retrieval
876
+ name: MTEB T2Retrieval
877
+ config: default
878
+ split: dev
879
+ revision: None
880
+ metrics:
881
+ - type: map_at_1
882
+ value: 28.29
883
+ - type: map_at_10
884
+ value: 78.441
885
+ - type: map_at_100
886
+ value: 82.043
887
+ - type: map_at_1000
888
+ value: 82.10499999999999
889
+ - type: map_at_3
890
+ value: 55.448
891
+ - type: map_at_5
892
+ value: 67.982
893
+ - type: mrr_at_1
894
+ value: 91.18
895
+ - type: mrr_at_10
896
+ value: 93.498
897
+ - type: mrr_at_100
898
+ value: 93.57
899
+ - type: mrr_at_1000
900
+ value: 93.572
901
+ - type: mrr_at_3
902
+ value: 93.112
903
+ - type: mrr_at_5
904
+ value: 93.351
905
+ - type: ndcg_at_1
906
+ value: 91.18
907
+ - type: ndcg_at_10
908
+ value: 85.849
909
+ - type: ndcg_at_100
910
+ value: 89.32600000000001
911
+ - type: ndcg_at_1000
912
+ value: 89.9
913
+ - type: ndcg_at_3
914
+ value: 87.333
915
+ - type: ndcg_at_5
916
+ value: 85.91499999999999
917
+ - type: precision_at_1
918
+ value: 91.18
919
+ - type: precision_at_10
920
+ value: 42.315000000000005
921
+ - type: precision_at_100
922
+ value: 5.029
923
+ - type: precision_at_1000
924
+ value: 0.517
925
+ - type: precision_at_3
926
+ value: 76.12400000000001
927
+ - type: precision_at_5
928
+ value: 63.690000000000005
929
+ - type: recall_at_1
930
+ value: 28.29
931
+ - type: recall_at_10
932
+ value: 84.679
933
+ - type: recall_at_100
934
+ value: 95.952
935
+ - type: recall_at_1000
936
+ value: 98.821
937
+ - type: recall_at_3
938
+ value: 56.987
939
+ - type: recall_at_5
940
+ value: 71.15599999999999
941
+ - task:
942
+ type: Classification
943
+ dataset:
944
+ type: C-MTEB/TNews-classification
945
+ name: MTEB TNews
946
+ config: default
947
+ split: validation
948
+ revision: None
949
+ metrics:
950
+ - type: accuracy
951
+ value: 53.09799999999999
952
+ - type: f1
953
+ value: 51.397192036892314
954
+ - task:
955
+ type: Clustering
956
+ dataset:
957
+ type: C-MTEB/ThuNewsClusteringP2P
958
+ name: MTEB ThuNewsClusteringP2P
959
+ config: default
960
+ split: test
961
+ revision: None
962
+ metrics:
963
+ - type: v_measure
964
+ value: 70.59693805158501
965
+ - task:
966
+ type: Clustering
967
+ dataset:
968
+ type: C-MTEB/ThuNewsClusteringS2S
969
+ name: MTEB ThuNewsClusteringS2S
970
+ config: default
971
+ split: test
972
+ revision: None
973
+ metrics:
974
+ - type: v_measure
975
+ value: 63.21127290121542
976
+ - task:
977
+ type: Retrieval
978
+ dataset:
979
+ type: C-MTEB/VideoRetrieval
980
+ name: MTEB VideoRetrieval
981
+ config: default
982
+ split: dev
983
+ revision: None
984
+ metrics:
985
+ - type: map_at_1
986
+ value: 61.3
987
+ - type: map_at_10
988
+ value: 70.658
989
+ - type: map_at_100
990
+ value: 71.096
991
+ - type: map_at_1000
992
+ value: 71.108
993
+ - type: map_at_3
994
+ value: 69.15
995
+ - type: map_at_5
996
+ value: 70.125
997
+ - type: mrr_at_1
998
+ value: 61.3
999
+ - type: mrr_at_10
1000
+ value: 70.658
1001
+ - type: mrr_at_100
1002
+ value: 71.096
1003
+ - type: mrr_at_1000
1004
+ value: 71.108
1005
+ - type: mrr_at_3
1006
+ value: 69.15
1007
+ - type: mrr_at_5
1008
+ value: 70.125
1009
+ - type: ndcg_at_1
1010
+ value: 61.3
1011
+ - type: ndcg_at_10
1012
+ value: 74.71
1013
+ - type: ndcg_at_100
1014
+ value: 76.783
1015
+ - type: ndcg_at_1000
1016
+ value: 77.09899999999999
1017
+ - type: ndcg_at_3
1018
+ value: 71.634
1019
+ - type: ndcg_at_5
1020
+ value: 73.399
1021
+ - type: precision_at_1
1022
+ value: 61.3
1023
+ - type: precision_at_10
1024
+ value: 8.72
1025
+ - type: precision_at_100
1026
+ value: 0.967
1027
+ - type: precision_at_1000
1028
+ value: 0.099
1029
+ - type: precision_at_3
1030
+ value: 26.267000000000003
1031
+ - type: precision_at_5
1032
+ value: 16.619999999999997
1033
+ - type: recall_at_1
1034
+ value: 61.3
1035
+ - type: recall_at_10
1036
+ value: 87.2
1037
+ - type: recall_at_100
1038
+ value: 96.7
1039
+ - type: recall_at_1000
1040
+ value: 99.2
1041
+ - type: recall_at_3
1042
+ value: 78.8
1043
+ - type: recall_at_5
1044
+ value: 83.1
1045
+ - task:
1046
+ type: Classification
1047
+ dataset:
1048
+ type: C-MTEB/waimai-classification
1049
+ name: MTEB Waimai
1050
+ config: default
1051
+ split: test
1052
+ revision: None
1053
+ metrics:
1054
+ - type: accuracy
1055
+ value: 88.01
1056
+ - type: ap
1057
+ value: 72.51537272974005
1058
+ - type: f1
1059
+ value: 86.49546025793478
1060
  ---
1061
+
1062
+
1063
+ **新闻 | News**
1064
+ **[2024-04-??]** stella-v4系列预计四月份发布，**专门针对检索和语义匹配任务，更多的考虑泛化性和私有通用测试集效果，向量维度可变，中英双语
1065
+ **。
1066
+ **[2024-02-27]** 开源stella-mrl-large-zh-v3.5-1792d模型，支持**向量可变维度**。
1067
+ **[2024-02-17]** 开源stella v3系列、dialogue编码模型和相关训练数据。
1068
+ **[2023-10-19]** 开源stella-base-en-v2 使用简单，**不需要任何前缀文本**。
1069
+ **[2023-10-12]** 开源stella-base-zh-v2和stella-large-zh-v2, 效果更好且使用简单，**不需要任何前缀文本**。
1070
+ **[2023-09-11]** 开源stella-base-zh和stella-large-zh
1071
+
1072
+ 欢迎去[本人主页](https://huggingface.co/infgrad)查看最新模型，并提出您的宝贵意见！
1073
+
1074
+ # 1 开源模型
1075
+
1076
+ 本次开源stella-mrl-large-zh-v3.5-1792d模型，
1077
+ 本模型是在stella-large-zh-v3-1792d的基础上使用[MRL](https://arxiv.org/abs/2205.13147)方法训练而成。
1078
+ 其主要特点是**可变的向量维度**。
1079
+
1080
+ # 2 使用方法
1081
+
1082
+ ```python
1083
+ from sentence_transformers import SentenceTransformer
1084
+ from sklearn.preprocessing import normalize
1085
+
1086
+ model = SentenceTransformer("infgrad/stella-mrl-large-zh-v3.5-1792d")
1087
+ # 注意先不要normalize! 选取前n维后再normalize
1088
+ vectors = model.encode(["text1", "text2"], normalize_embeddings=False)
1089
+ print(vectors.shape) # shape is [2,1792]
1090
+ # n_dims越大效果越好，但是时空消耗就越大。建议维度选取128的倍数，因为是这么训练的
1091
+ n_dims = 768
1092
+ cut_vecs = normalize(vectors[:, :n_dims])
1093
+
1094
+ ```
1095
+
1096
+ # 3 不同向量维度的CMTEB得分
1097
+
1098
+ stella-mrl-large-zh-v3.5-1792d_1024 代表取前1024维。整体趋势是维度越大效果越好。
1099
+
1100
+ | Model | Retrieval | STS | PairClassification | Classification | Reranking | Clustering | CMTEB-Score |
1101
+ |:------------------------------------|:---------:|:-----:|:------------------:|:--------------:|:---------:|:----------:|:-----------:|
1102
+ | stella-mrl-large-zh-v3.5-1792d_128 | 70.01 | 62.17 | 87.99 | 70.67 | 66.77 | 53.55 | 67.16 |
1103
+ | stella-mrl-large-zh-v3.5-1792d_256 | 72.19 | 62.41 | 88.09 | 71.22 | 68.32 | 53.38 | 68.02 |
1104
+ | stella-mrl-large-zh-v3.5-1792d_384 | 72.77 | 62.43 | 88.26 | 71.34 | 68.31 | 53.87 | 68.25 |
1105
+ | stella-mrl-large-zh-v3.5-1792d_512 | 73.11 | 62.45 | 88.16 | 71.46 | 68.32 | 53.28 | 68.29 |
1106
+ | stella-mrl-large-zh-v3.5-1792d_640 | 73.27 | 62.49 | 88.21 | 71.46 | 68.69 | 53.63 | 68.42 |
1107
+ | stella-mrl-large-zh-v3.5-1792d_768 | 73.38 | 62.5 | 88.19 | 71.49 | 68.64 | 53.77 | 68.47 |
1108
+ | stella-mrl-large-zh-v3.5-1792d_896 | 73.37 | 62.5 | 88.14 | 71.51 | 68.44 | 54.13 | 68.49 |
1109
+ | stella-mrl-large-zh-v3.5-1792d_1024 | 73.43 | 62.51 | 88.16 | 71.52 | 68.59 | 53.43 | 68.44 |
1110
+ | stella-mrl-large-zh-v3.5-1792d_1152 | 73.46 | 62.49 | 88.16 | 71.57 | 68.55 | 53.67 | 68.49 |
1111
+ | stella-mrl-large-zh-v3.5-1792d_1280 | 73.48 | 62.51 | 88.12 | 71.55 | 68.44 | 53.74 | 68.48 |
1112
+ | stella-mrl-large-zh-v3.5-1792d_1408 | 73.48 | 62.51 | 88.14 | 71.58 | 68.46 | 53.69 | 68.48 |
1113
+ | stella-mrl-large-zh-v3.5-1792d_1536 | 73.49 | 62.5 | 88.11 | 71.55 | 68.5 | 54.06 | 68.52 |
1114
+ | stella-mrl-large-zh-v3.5-1792d_1664 | 73.56 | 62.49 | 88.06 | 71.56 | 68.47 | 54.28 | 68.56 |
1115
+ | stella-mrl-large-zh-v3.5-1792d_1792 | 73.51 | 62.48 | 88.09 | 71.56 | 68.45 | 54.39 | 68.56 |
1116
+
1117
+ 上述表格中stella-mrl-large-zh-v3.5-1792d_1792的得分为68.56和榜单68.55得分不一致，原因和权重类型���关，小差异请忽略不计。