prithivida
commited on
Commit
•
b2893b9
1
Parent(s):
ab4e07e
Update README.md
Browse files
README.md
CHANGED
@@ -29,19 +29,12 @@ SPLADE models are a fine balance between retrieval effectiveness (quality) and r
|
|
29 |
*(Pure MLE folks should not conflate efficiency to model inference efficiency. Our main focus is on retrieval efficiency. Hereinafter efficiency is a short hand for retrieval efficiency unless explicitly qualified otherwise. Not that inference efficiency is not important, we will address that subsequently.)*
|
30 |
|
31 |
**TL;DR of Our attempt & results**
|
32 |
-
1. FLOPS tuning:
|
33 |
-
|
34 |
-
|
35 |
-
|
36 |
-
|
37 |
-
|
38 |
-
2. and a retrieval latency of - **47.27ms**. (multi-threaded)
|
39 |
-
3. On **mono-GPU** with **only 5 negatives per query**.
|
40 |
-
4. For Industry setting
|
41 |
-
- Effectiveness on custom domains needs more than just **Trading FLOPS for tiny gains**.
|
42 |
-
- The Premise "SPLADE++ are not well suited to mono-cpu retrieval" does not hold.
|
43 |
-
|
44 |
-
<img src="./ID.png" width=500 height=350/>
|
45 |
|
46 |
*Note: The paper refers to the best performing models as SPLADE++, hence for consistency we are reusing the same.*
|
47 |
|
|
|
29 |
*(Pure MLE folks should not conflate efficiency to model inference efficiency. Our main focus is on retrieval efficiency. Hereinafter efficiency is a short hand for retrieval efficiency unless explicitly qualified otherwise. Not that inference efficiency is not important, we will address that subsequently.)*
|
30 |
|
31 |
**TL;DR of Our attempt & results**
|
32 |
+
1. FLOPS tuning: Seperate **Seq lens and Severely restrictive token budget** doc(128) & query(24) NOT 256 unlike Official SPLADE++. Inspired from **SparseEmbed** (instead of 2 models for query & doc).
|
33 |
+
3. Init Weights: **MLM adapted on MS MARCO corpus**.
|
34 |
+
4. Achieves a modest yet competitive effectiveness **MRR@10 37.22** in ID data (& OOD) and a retrieval latency of - **47.27ms**. (multi-threaded) all On **mono-GPU** with **only 5 negatives per query**.
|
35 |
+
4. For Industry setting: Effectiveness on custom domains needs more than just **Trading FLOPS for tiny gains** and The Premise "SPLADE++ are not well suited to mono-cpu retrieval" does not hold.
|
36 |
+
|
37 |
+
<img src="./ID.png" width=550 height=450/>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
38 |
|
39 |
*Note: The paper refers to the best performing models as SPLADE++, hence for consistency we are reusing the same.*
|
40 |
|