Commit
•
345e984
1
Parent(s):
2ae59f1
Update README.md
Browse files
README.md
CHANGED
|
@@ -264,7 +264,59 @@ embeddings = model.encode(['How is the weather today?'], task_type='retrieval.qu
|
|
| 264 |
|
| 265 |
## Performance
|
| 266 |
|
| 267 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 268 |
|
| 269 |
## Contact
|
| 270 |
|
|
|
|
| 264 |
|
| 265 |
## Performance
|
| 266 |
|
| 267 |
+
### English MTEB
|
| 268 |
+
| Model | Average | CF | CL | PC | RR | RT | STS | SM |
|
| 269 |
+
|:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|
| 270 |
+
| jina-embeddings-v2-en | 58.12 | 68.82| 40.08| 84.44| 55.09| 45.64| 80.00| 30.56|
|
| 271 |
+
| jina-embeddings-v3 | **65.60** | **82.58**| 45.27| 84.01| 58.13| 53.87| **85.8** | 30.98|
|
| 272 |
+
| text-embedding-3-large | 62.03 | 75.45| 49.01| 84.22| 59.16| 55.44| 81.04| 29.92|
|
| 273 |
+
| multilingual-e5-large-instruct | 64.41 | 77.56| 47.1 | 86.19| 58.58| 52.47| 84.78| 30.39|
|
| 274 |
+
| Cohere-embed-multilingual-v3.0 | 60.08 | 64.01| 46.6 | 86.15| 57.86| 53.84| 83.15| 30.99|
|
| 275 |
+
|
| 276 |
+
### Multilingual MTEB
|
| 277 |
+
|
| 278 |
+
| Model | Average | CF | CL | PC | RR | RT | STS | SM |
|
| 279 |
+
|:------------------------------:|:-------:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
|
| 280 |
+
| jina-embeddings-v2* | 60.54 | 65.69| 39.36| **82.95**| 66.57| 58.24| 66.6 | - |
|
| 281 |
+
| jina-embeddings-v3 | **64.44** | **71.46**| 46.71| 76.91| 63.98| 57.98| **69.83**| - |
|
| 282 |
+
| multilingual-e5-large | 59.58 | 65.22| 42.12| 76.95| 63.4 | 52.37| 64.65| - |
|
| 283 |
+
| multilingual-e5-large-instruct | 64.25 | 67.45| **52.12**| 77.79| **69.02**| **58.38**| 68.77| - |
|
| 284 |
+
|
| 285 |
+
**Notes:**
|
| 286 |
+
- CF: Classification Accuracy
|
| 287 |
+
- CL: Clustering V measure
|
| 288 |
+
- PC: Pair Classification Average Precision
|
| 289 |
+
- RR: Reranking MAP
|
| 290 |
+
- RT: Retrieval nDCG@10
|
| 291 |
+
- STS: Sentence Similarity Spearman Correlation
|
| 292 |
+
- SM: Summarization Spearman Correlation
|
| 293 |
+
- `*`: Jina Embeddings v2 bilingual model suite, only evaluated on Chinese, Spanish and German tasks.
|
| 294 |
+
|
| 295 |
+
### Long Context Tasks (LongEmbed)
|
| 296 |
+
|
| 297 |
+
| Model | Average | NarrativeQA | Needle | Passkey | QMSum | SummScreen | WikiQA |
|
| 298 |
+
|:--------------------:|:-------:|:-----------:|:------:|:-------:|:-----:|:----------:|:------:|
|
| 299 |
+
| jina-embeddings-v3* | **70.39** | 33.32 | **84.00** | **100.00** | **39.75** | 92.78 | 72.46 |
|
| 300 |
+
| jina-embeddings-v2 | 58.12 | 37.89 | 54.25 | 50.25 | 38.87 | 93.48 | 73.99 |
|
| 301 |
+
| text-embedding-3-large | 51.3 | 44.09 | 29.25 | 63.00 | 32.49 | 84.80 | 54.16 |
|
| 302 |
+
| baai-bge-m3 | 56.56 | **45.76** | 40.25 | 46.00 | 35.54 | **94.09** | **77.73** |
|
| 303 |
+
|
| 304 |
+
**Notes:**
|
| 305 |
+
- `*`: text-matching adapter
|
| 306 |
+
|
| 307 |
+
|
| 308 |
+
#### Matryoshka Embeddings
|
| 309 |
+
|
| 310 |
+
| MRL Dimension | Retrieval | STS |
|
| 311 |
+
|:-------------:|:---------:|:----:|
|
| 312 |
+
| 32 | 52.54 | 76.35|
|
| 313 |
+
| 64 | 58.54 | 77.03|
|
| 314 |
+
| 128 | 61.64 | 77.43|
|
| 315 |
+
| 256 | 62.72 | 77.56|
|
| 316 |
+
| 512 | 63.16 | 77.59|
|
| 317 |
+
| 768 | 63.30 | 77.59|
|
| 318 |
+
| 1024 | 63.35 | 77.58|
|
| 319 |
+
|
| 320 |
|
| 321 |
## Contact
|
| 322 |
|