Update README.md
Browse files
README.md
CHANGED
@@ -138,14 +138,14 @@ Note that both ColBERTv2 and Jina-ColBERT-v1 only employ MSMARCO passage ranking
|
|
138 |
|
139 |
We also evaluate the zero-shot performance on datasets where documents have longer context length and compare with some long-context embedding models. Here we use the [LoCo benchmark](https://www.together.ai/blog/long-context-retrieval-models-with-monarch-mixer), which contains 5 datasets with long context length.
|
140 |
|
141 |
-
| Model |
|
142 |
| --- | :---: | :---: | :---: |
|
143 |
-
| ColBERTv2 |
|
144 |
-
| Jina-ColBERT-v1 |
|
145 |
-
| Jina-ColBERT-v1 |
|
146 |
-
| Jina-embeddings-v2-base-en |
|
147 |
|
148 |
-
\* denotes that we
|
149 |
|
150 |
**To summarize, Jina-ColBERT achieves the comparable performance with ColBERTv2 on all benchmarks, and outperforms ColBERTv2 on datasets in where documents have longer context length.**
|
151 |
|
|
|
138 |
|
139 |
We also evaluate the zero-shot performance on datasets where documents have longer context length and compare with some long-context embedding models. Here we use the [LoCo benchmark](https://www.together.ai/blog/long-context-retrieval-models-with-monarch-mixer), which contains 5 datasets with long context length.
|
140 |
|
141 |
+
| Model | Used context length | Model max context length | Avg. NDCG@10 |
|
142 |
| --- | :---: | :---: | :---: |
|
143 |
+
| ColBERTv2 | 512 | 512 | 74.3 |
|
144 |
+
| Jina-ColBERT-v1 (truncated) | 512* | 8192 | 75.5 |
|
145 |
+
| Jina-ColBERT-v1 | 8192 | 8192 | 83.7 |
|
146 |
+
| Jina-embeddings-v2-base-en | 8192 | 8192 | **85.4** |
|
147 |
|
148 |
+
\* denotes that we truncate the context length to the length of 512 for document but the query length is still 512.
|
149 |
|
150 |
**To summarize, Jina-ColBERT achieves the comparable performance with ColBERTv2 on all benchmarks, and outperforms ColBERTv2 on datasets in where documents have longer context length.**
|
151 |
|