jupyterjazz commited on
Commit
b04e355
1 Parent(s): 8bf1330

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -12
README.md CHANGED
@@ -270,7 +270,7 @@ embeddings = model.encode(['How is the weather today?'], task_type='retrieval.qu
270
  ## Performance
271
 
272
  ### English MTEB
273
- | Model | Average | CF | CL | PC | RR | RT | STS | SM |
274
  |:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
275
  | jina-embeddings-v2-en | 58.12 | 68.82| 40.08| 84.44| 55.09| 45.64| 80.00| 30.56|
276
  | jina-embeddings-v3 | **65.60** | **82.58**| 45.27| 84.01| 58.13| 53.87| **85.8** | 30.98|
@@ -280,22 +280,13 @@ embeddings = model.encode(['How is the weather today?'], task_type='retrieval.qu
280
 
281
  ### Multilingual MTEB
282
 
283
- | Model | Average | CF | CL | PC | RR | RT | STS | SM |
284
  |:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
285
- | jina-embeddings-v2* | 60.54 | 65.69| 39.36| **82.95**| 66.57| 58.24| 66.6 | - |
286
  | jina-embeddings-v3 | **64.44** | **71.46**| 46.71| 76.91| 63.98| 57.98| **69.83**| - |
287
  | multilingual-e5-large | 59.58 | 65.22| 42.12| 76.95| 63.4 | 52.37| 64.65| - |
288
  | multilingual-e5-large-instruct | 64.25 | 67.45| **52.12**| 77.79| **69.02**| **58.38**| 68.77| - |
289
 
290
- **Notes:**
291
- - CF: Classification Accuracy
292
- - CL: Clustering V measure
293
- - PC: Pair Classification Average Precision
294
- - RR: Reranking MAP
295
- - RT: Retrieval nDCG@10
296
- - STS: Sentence Similarity Spearman Correlation
297
- - SM: Summarization Spearman Correlation
298
- - `*`: Jina Embeddings v2 bilingual model suite, only evaluated on Chinese, Spanish and German tasks.
299
 
300
  ### Long Context Tasks (LongEmbed)
301
 
@@ -317,6 +308,7 @@ embeddings = model.encode(['How is the weather today?'], task_type='retrieval.qu
317
  | Retrieval | 52.54| 58.54| 61.64| 62.72| 63.16| 63.30| 63.35|
318
  | STS | 76.35| 77.03| 77.43| 77.56| 77.59| 77.59| 77.58|
319
 
 
320
 
321
  ## Contact
322
 
 
270
  ## Performance
271
 
272
  ### English MTEB
273
+ | Model | Average | Classification | Clustering | Pair Classification | Reranking | Retrieval | STS | Summarization |
274
  |:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
275
  | jina-embeddings-v2-en | 58.12 | 68.82| 40.08| 84.44| 55.09| 45.64| 80.00| 30.56|
276
  | jina-embeddings-v3 | **65.60** | **82.58**| 45.27| 84.01| 58.13| 53.87| **85.8** | 30.98|
 
280
 
281
  ### Multilingual MTEB
282
 
283
+ | Model | Average | Classification | Clustering | Pair Classification | Reranking | Retrieval | STS | Summarization |
284
  |:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
285
+ | jina-embeddings-v2 | 60.54 | 65.69| 39.36| **82.95**| 66.57| 58.24| 66.6 | - |
286
  | jina-embeddings-v3 | **64.44** | **71.46**| 46.71| 76.91| 63.98| 57.98| **69.83**| - |
287
  | multilingual-e5-large | 59.58 | 65.22| 42.12| 76.95| 63.4 | 52.37| 64.65| - |
288
  | multilingual-e5-large-instruct | 64.25 | 67.45| **52.12**| 77.79| **69.02**| **58.38**| 68.77| - |
289
 
 
 
 
 
 
 
 
 
 
290
 
291
  ### Long Context Tasks (LongEmbed)
292
 
 
308
  | Retrieval | 52.54| 58.54| 61.64| 62.72| 63.16| 63.30| 63.35|
309
  | STS | 76.35| 77.03| 77.43| 77.56| 77.59| 77.59| 77.58|
310
 
311
+ For a comprehensive evaluation and detailed metrics, please refer to the full paper available here (coming soon).
312
 
313
  ## Contact
314