Hugging Face's logo

Forbu14
/
openai_clip_embeddings

mteb
Eval Results
Model card Files Community
Forbu14 commited on
Commit
00c32ba
•
1 Parent(s): 69d022a

adding performance on MTEB dataset for clip embeddings

Browse files
Files changed (3) hide show
  1. Dockerfile +4 -1
  2. README.md +175 -1
  3. mteb_metadata.md +163 -0
Dockerfile CHANGED
@@ -1,4 +1,7 @@
1
- FROM huggingface/transformers-pytorch-cpu:latest
 
 
 
2
 
3
  # install requirements
4
  COPY requirements.txt .
 
1
+ FROM pytorch/pytorch:latest
2
+
3
+ # install git
4
+ RUN apt-get update && apt-get install -y git
5
 
6
  # install requirements
7
  COPY requirements.txt .
README.md CHANGED
@@ -1,3 +1,177 @@
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ tags:
3
+ - mteb
4
+ model-index:
5
+ - name: json_results
6
+ results:
7
+ - task:
8
+ type: Classification
9
+ dataset:
10
+ type: mteb/amazon_counterfactual
11
+ name: MTEB AmazonCounterfactualClassification (en)
12
+ config: en
13
+ split: test
14
+ revision: e8379541af4e31359cca9fbcf4b00f2671dba205
15
+ metrics:
16
+ - type: accuracy
17
+ value: 57.49253731343285
18
+ - type: ap
19
+ value: 23.59442736353998
20
+ - type: f1
21
+ value: 52.20223389089595
22
+ - task:
23
+ type: Classification
24
+ dataset:
25
+ type: mteb/amazon_reviews_multi
26
+ name: MTEB AmazonReviewsClassification (en)
27
+ config: en
28
+ split: test
29
+ revision: 1399c76144fd37290681b995c656ef9b2e06e26d
30
+ metrics:
31
+ - type: accuracy
32
+ value: 30.59
33
+ - type: f1
34
+ value: 30.418224700389747
35
+ - task:
36
+ type: Classification
37
+ dataset:
38
+ type: mteb/banking77
39
+ name: MTEB Banking77Classification
40
+ config: default
41
+ split: test
42
+ revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
43
+ metrics:
44
+ - type: accuracy
45
+ value: 73.41883116883116
46
+ - type: f1
47
+ value: 73.3645582123564
48
+ - task:
49
+ type: Clustering
50
+ dataset:
51
+ type: mteb/biorxiv-clustering-p2p
52
+ name: MTEB BiorxivClusteringP2P
53
+ config: default
54
+ split: test
55
+ revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40
56
+ metrics:
57
+ - type: v_measure
58
+ value: 29.33118844069676
59
+ - task:
60
+ type: Clustering
61
+ dataset:
62
+ type: mteb/biorxiv-clustering-s2s
63
+ name: MTEB BiorxivClusteringS2S
64
+ config: default
65
+ split: test
66
+ revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908
67
+ metrics:
68
+ - type: v_measure
69
+ value: 27.812326878347093
70
+ - task:
71
+ type: Classification
72
+ dataset:
73
+ type: mteb/emotion
74
+ name: MTEB EmotionClassification
75
+ config: default
76
+ split: test
77
+ revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
78
+ metrics:
79
+ - type: accuracy
80
+ value: 33.62
81
+ - type: f1
82
+ value: 29.639357232727388
83
+ - task:
84
+ type: Classification
85
+ dataset:
86
+ type: mteb/imdb
87
+ name: MTEB ImdbClassification
88
+ config: default
89
+ split: test
90
+ revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
91
+ metrics:
92
+ - type: accuracy
93
+ value: 56.1716
94
+ - type: ap
95
+ value: 53.588732808885574
96
+ - type: f1
97
+ value: 55.863727214981004
98
+ - task:
99
+ type: Classification
100
+ dataset:
101
+ type: mteb/mtop_domain
102
+ name: MTEB MTOPDomainClassification (en)
103
+ config: en
104
+ split: test
105
+ revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
106
+ metrics:
107
+ - type: accuracy
108
+ value: 87.07250341997262
109
+ - type: f1
110
+ value: 86.63685613523198
111
+ - task:
112
+ type: Classification
113
+ dataset:
114
+ type: mteb/mtop_intent
115
+ name: MTEB MTOPIntentClassification (en)
116
+ config: en
117
+ split: test
118
+ revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
119
+ metrics:
120
+ - type: accuracy
121
+ value: 61.95622435020519
122
+ - type: f1
123
+ value: 41.66240550937103
124
+ - task:
125
+ type: Classification
126
+ dataset:
127
+ type: mteb/amazon_massive_intent
128
+ name: MTEB MassiveIntentClassification (en)
129
+ config: en
130
+ split: test
131
+ revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
132
+ metrics:
133
+ - type: accuracy
134
+ value: 62.96234028244788
135
+ - type: f1
136
+ value: 60.20385917259002
137
+ - task:
138
+ type: Classification
139
+ dataset:
140
+ type: mteb/amazon_massive_scenario
141
+ name: MTEB MassiveScenarioClassification (en)
142
+ config: en
143
+ split: test
144
+ revision: 7d571f92784cd94a019292a1f45445077d0ef634
145
+ metrics:
146
+ - type: accuracy
147
+ value: 71.46603900470747
148
+ - type: f1
149
+ value: 70.96623988750936
150
+ - task:
151
+ type: Classification
152
+ dataset:
153
+ type: mteb/tweet_sentiment_extraction
154
+ name: MTEB TweetSentimentExtractionClassification
155
+ config: default
156
+ split: test
157
+ revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
158
+ metrics:
159
+ - type: accuracy
160
+ value: 49.34352009054896
161
+ - type: f1
162
+ value: 49.58635289569058
163
  ---
164
+
165
+ license: apache-2.0
166
+
167
+
168
+ Here we estimate the performance of the CLIP embeddings (contrastive training between text - image data).
169
+
170
+
171
+
172
+
173
+
174
+
175
+
176
+
177
+
mteb_metadata.md ADDED
@@ -0,0 +1,163 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ tags:
3
+ - mteb
4
+ model-index:
5
+ - name: json_results
6
+ results:
7
+ - task:
8
+ type: Classification
9
+ dataset:
10
+ type: mteb/amazon_counterfactual
11
+ name: MTEB AmazonCounterfactualClassification (en)
12
+ config: en
13
+ split: test
14
+ revision: e8379541af4e31359cca9fbcf4b00f2671dba205
15
+ metrics:
16
+ - type: accuracy
17
+ value: 57.49253731343285
18
+ - type: ap
19
+ value: 23.59442736353998
20
+ - type: f1
21
+ value: 52.20223389089595
22
+ - task:
23
+ type: Classification
24
+ dataset:
25
+ type: mteb/amazon_reviews_multi
26
+ name: MTEB AmazonReviewsClassification (en)
27
+ config: en
28
+ split: test
29
+ revision: 1399c76144fd37290681b995c656ef9b2e06e26d
30
+ metrics:
31
+ - type: accuracy
32
+ value: 30.59
33
+ - type: f1
34
+ value: 30.418224700389747
35
+ - task:
36
+ type: Classification
37
+ dataset:
38
+ type: mteb/banking77
39
+ name: MTEB Banking77Classification
40
+ config: default
41
+ split: test
42
+ revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
43
+ metrics:
44
+ - type: accuracy
45
+ value: 73.41883116883116
46
+ - type: f1
47
+ value: 73.3645582123564
48
+ - task:
49
+ type: Clustering
50
+ dataset:
51
+ type: mteb/biorxiv-clustering-p2p
52
+ name: MTEB BiorxivClusteringP2P
53
+ config: default
54
+ split: test
55
+ revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40
56
+ metrics:
57
+ - type: v_measure
58
+ value: 29.33118844069676
59
+ - task:
60
+ type: Clustering
61
+ dataset:
62
+ type: mteb/biorxiv-clustering-s2s
63
+ name: MTEB BiorxivClusteringS2S
64
+ config: default
65
+ split: test
66
+ revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908
67
+ metrics:
68
+ - type: v_measure
69
+ value: 27.812326878347093
70
+ - task:
71
+ type: Classification
72
+ dataset:
73
+ type: mteb/emotion
74
+ name: MTEB EmotionClassification
75
+ config: default
76
+ split: test
77
+ revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
78
+ metrics:
79
+ - type: accuracy
80
+ value: 33.62
81
+ - type: f1
82
+ value: 29.639357232727388
83
+ - task:
84
+ type: Classification
85
+ dataset:
86
+ type: mteb/imdb
87
+ name: MTEB ImdbClassification
88
+ config: default
89
+ split: test
90
+ revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
91
+ metrics:
92
+ - type: accuracy
93
+ value: 56.1716
94
+ - type: ap
95
+ value: 53.588732808885574
96
+ - type: f1
97
+ value: 55.863727214981004
98
+ - task:
99
+ type: Classification
100
+ dataset:
101
+ type: mteb/mtop_domain
102
+ name: MTEB MTOPDomainClassification (en)
103
+ config: en
104
+ split: test
105
+ revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
106
+ metrics:
107
+ - type: accuracy
108
+ value: 87.07250341997262
109
+ - type: f1
110
+ value: 86.63685613523198
111
+ - task:
112
+ type: Classification
113
+ dataset:
114
+ type: mteb/mtop_intent
115
+ name: MTEB MTOPIntentClassification (en)
116
+ config: en
117
+ split: test
118
+ revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
119
+ metrics:
120
+ - type: accuracy
121
+ value: 61.95622435020519
122
+ - type: f1
123
+ value: 41.66240550937103
124
+ - task:
125
+ type: Classification
126
+ dataset:
127
+ type: mteb/amazon_massive_intent
128
+ name: MTEB MassiveIntentClassification (en)
129
+ config: en
130
+ split: test
131
+ revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
132
+ metrics:
133
+ - type: accuracy
134
+ value: 62.96234028244788
135
+ - type: f1
136
+ value: 60.20385917259002
137
+ - task:
138
+ type: Classification
139
+ dataset:
140
+ type: mteb/amazon_massive_scenario
141
+ name: MTEB MassiveScenarioClassification (en)
142
+ config: en
143
+ split: test
144
+ revision: 7d571f92784cd94a019292a1f45445077d0ef634
145
+ metrics:
146
+ - type: accuracy
147
+ value: 71.46603900470747
148
+ - type: f1
149
+ value: 70.96623988750936
150
+ - task:
151
+ type: Classification
152
+ dataset:
153
+ type: mteb/tweet_sentiment_extraction
154
+ name: MTEB TweetSentimentExtractionClassification
155
+ config: default
156
+ split: test
157
+ revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
158
+ metrics:
159
+ - type: accuracy
160
+ value: 49.34352009054896
161
+ - type: f1
162
+ value: 49.58635289569058
163
+ ---