Hugging Face's logo

tomaarsen
/
ModernBERT-base-gooaq

Sentence Similarity
sentence-transformers
Safetensors
English
modernbert
feature-extraction
Generated from Trainer
dataset_size:3012496
loss:CachedMultipleNegativesRankingLoss
Eval Results
Inference Endpoints
Model card Files Community
tomaarsen HF staff commited on
Commit
8b67887
•
1 Parent(s): 08d82cb

Add new SentenceTransformer model

Browse files
Files changed (10) hide show
  1. 1_Pooling/config.json +10 -0
  2. README.md +802 -0
  3. config.json +47 -0
  4. config_sentence_transformers.json +10 -0
  5. model.safetensors +3 -0
  6. modules.json +14 -0
  7. sentence_bert_config.json +4 -0
  8. special_tokens_map.json +37 -0
  9. tokenizer.json +0 -0
  10. tokenizer_config.json +945 -0
1_Pooling/config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "word_embedding_dimension": 768,
3
+ "pooling_mode_cls_token": false,
4
+ "pooling_mode_mean_tokens": true,
5
+ "pooling_mode_max_tokens": false,
6
+ "pooling_mode_mean_sqrt_len_tokens": false,
7
+ "pooling_mode_weightedmean_tokens": false,
8
+ "pooling_mode_lasttoken": false,
9
+ "include_prompt": true
10
+ }
README.md ADDED
@@ -0,0 +1,802 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - sentence-transformers
6
+ - sentence-similarity
7
+ - feature-extraction
8
+ - generated_from_trainer
9
+ - dataset_size:3012496
10
+ - loss:CachedMultipleNegativesRankingLoss
11
+ base_model: answerdotai/ModernBERT-base
12
+ widget:
13
+ - source_sentence: how much is a car title transfer in minnesota?
14
+ sentences:
15
+ - This complex is a larger molecule than the original crystal violet stain and iodine
16
+ and is insoluble in water. ... Conversely, the the outer membrane of Gram negative
17
+ bacteria is degraded and the thinner peptidoglycan layer of Gram negative cells
18
+ is unable to retain the crystal violet-iodine complex and the color is lost.
19
+ - Get insurance on the car and provide proof. Bring this information (including
20
+ the title) to the Minnesota DVS office, as well as $10 for the filing fee and
21
+ $7.25 for the titling fee. There is also a $10 transfer tax, as well as a 6.5%
22
+ sales tax on the purchase price.
23
+ - 'One of the risks of DNP is that it accelerates the metabolism to a dangerously
24
+ fast level. Our metabolic system operates at the rate it does for a reason – it
25
+ is safe. Speeding up the metabolism may help burn off fat, but it can also trigger
26
+ a number of potentially dangerous side effects, such as: fever.'
27
+ - source_sentence: what is the difference between 18 and 20 inch tires?
28
+ sentences:
29
+ - The only real difference is a 20" rim would be more likely to be damaged, as you
30
+ pointed out. Beyond looks, there is zero benefit for the 20" rim. Also, just the
31
+ availability of tires will likely be much more limited for the larger rim. ...
32
+ Tire selection is better for 18" wheels than 20" wheels.
33
+ - '[''Open your Outlook app on your mobile device and click on the Settings gear
34
+ icon.'', ''Under Settings, click on the Signature option.'', ''Enter either a
35
+ generic signature that could be used for all email accounts tied to your Outlook
36
+ app, or a specific signature, Per Account Signature, for each email account.'']'
37
+ - The average normal body temperature is around 98.6 degrees Fahrenheit, or 37 degrees
38
+ Celsius. If your body temperature drops to just a few degrees lower than this,
39
+ your blood vessels in your hands, feet, arms, and legs start to get narrower.
40
+ - source_sentence: whom the bell tolls meaning?
41
+ sentences:
42
+ - 'Answer: Humans are depicted in Hindu art often in sensuous and erotic postures.'
43
+ - The phrase "For whom the bell tolls" refers to the church bells that are rung
44
+ when a person dies. Hence, the author is suggesting that we should not be curious
45
+ as to for whom the church bell is tolling for. It is for all of us.
46
+ - '[''Automatically.'', ''When connected to car Bluetooth and,'', ''Manually.'']'
47
+ - source_sentence: how long before chlamydia symptoms appear?
48
+ sentences:
49
+ - Most people who have chlamydia don't notice any symptoms. If you do get symptoms,
50
+ these usually appear between 1 and 3 weeks after having unprotected sex with an
51
+ infected person. For some people they don't develop until many months later. Sometimes
52
+ the symptoms can disappear after a few days.
53
+ - '[''Open the My Verizon app . ... '', ''Tap the Menu icon. ... '', ''Tap Manage
54
+ device for the appropriate mobile number. ... '', ''Tap Transfer content between
55
+ phones. ... '', ''Tap Start Transfer.'']'
56
+ - 'Psychiatrist vs Psychologist A psychiatrist is classed as a medical doctor, they
57
+ include a physical examination of symptoms in their assessment and are able to
58
+ prescribe medicine: a psychologist is also a doctor by virtue of their PHD level
59
+ qualification, but is not medically trained and cannot prescribe.'
60
+ - source_sentence: are you human korean novela?
61
+ sentences:
62
+ - Many cysts heal on their own, which means that conservative treatments like rest
63
+ and anti-inflammatory painkillers can often be enough to get rid of them. However,
64
+ in some cases, routine drainage of the sac may be necessary to reduce symptoms.
65
+ - A relative of European pear varieties like Bartlett and Anjou, the Asian pear
66
+ is great used in recipes or simply eaten out of hand. It retains a crispness that
67
+ works well in slaws and salads, and it holds its shape better than European pears
68
+ when baked and cooked.
69
+ - 'Are You Human? (Korean: 너도 인간이니; RR: Neodo Inganini; lit. Are You Human Too?)
70
+ is a 2018 South Korean television series starring Seo Kang-jun and Gong Seung-yeon.
71
+ It aired on KBS2''s Mondays and Tuesdays at 22:00 (KST) time slot, from June 4
72
+ to August 7, 2018.'
73
+ datasets:
74
+ - sentence-transformers/gooaq
75
+ pipeline_tag: sentence-similarity
76
+ library_name: sentence-transformers
77
+ metrics:
78
+ - cosine_accuracy@1
79
+ - cosine_accuracy@3
80
+ - cosine_accuracy@5
81
+ - cosine_accuracy@10
82
+ - cosine_precision@1
83
+ - cosine_precision@3
84
+ - cosine_precision@5
85
+ - cosine_precision@10
86
+ - cosine_recall@1
87
+ - cosine_recall@3
88
+ - cosine_recall@5
89
+ - cosine_recall@10
90
+ - cosine_ndcg@10
91
+ - cosine_mrr@10
92
+ - cosine_map@100
93
+ model-index:
94
+ - name: SentenceTransformer based on answerdotai/ModernBERT-base
95
+ results:
96
+ - task:
97
+ type: information-retrieval
98
+ name: Information Retrieval
99
+ dataset:
100
+ name: NanoNQ
101
+ type: NanoNQ
102
+ metrics:
103
+ - type: cosine_accuracy@1
104
+ value: 0.38
105
+ name: Cosine Accuracy@1
106
+ - type: cosine_accuracy@3
107
+ value: 0.64
108
+ name: Cosine Accuracy@3
109
+ - type: cosine_accuracy@5
110
+ value: 0.7
111
+ name: Cosine Accuracy@5
112
+ - type: cosine_accuracy@10
113
+ value: 0.8
114
+ name: Cosine Accuracy@10
115
+ - type: cosine_precision@1
116
+ value: 0.38
117
+ name: Cosine Precision@1
118
+ - type: cosine_precision@3
119
+ value: 0.22
120
+ name: Cosine Precision@3
121
+ - type: cosine_precision@5
122
+ value: 0.14400000000000002
123
+ name: Cosine Precision@5
124
+ - type: cosine_precision@10
125
+ value: 0.08199999999999999
126
+ name: Cosine Precision@10
127
+ - type: cosine_recall@1
128
+ value: 0.36
129
+ name: Cosine Recall@1
130
+ - type: cosine_recall@3
131
+ value: 0.62
132
+ name: Cosine Recall@3
133
+ - type: cosine_recall@5
134
+ value: 0.67
135
+ name: Cosine Recall@5
136
+ - type: cosine_recall@10
137
+ value: 0.74
138
+ name: Cosine Recall@10
139
+ - type: cosine_ndcg@10
140
+ value: 0.5673854489333459
141
+ name: Cosine Ndcg@10
142
+ - type: cosine_mrr@10
143
+ value: 0.5237460317460316
144
+ name: Cosine Mrr@10
145
+ - type: cosine_map@100
146
+ value: 0.5116785860647901
147
+ name: Cosine Map@100
148
+ - task:
149
+ type: information-retrieval
150
+ name: Information Retrieval
151
+ dataset:
152
+ name: NanoMSMARCO
153
+ type: NanoMSMARCO
154
+ metrics:
155
+ - type: cosine_accuracy@1
156
+ value: 0.32
157
+ name: Cosine Accuracy@1
158
+ - type: cosine_accuracy@3
159
+ value: 0.56
160
+ name: Cosine Accuracy@3
161
+ - type: cosine_accuracy@5
162
+ value: 0.66
163
+ name: Cosine Accuracy@5
164
+ - type: cosine_accuracy@10
165
+ value: 0.82
166
+ name: Cosine Accuracy@10
167
+ - type: cosine_precision@1
168
+ value: 0.32
169
+ name: Cosine Precision@1
170
+ - type: cosine_precision@3
171
+ value: 0.18666666666666665
172
+ name: Cosine Precision@3
173
+ - type: cosine_precision@5
174
+ value: 0.132
175
+ name: Cosine Precision@5
176
+ - type: cosine_precision@10
177
+ value: 0.08199999999999999
178
+ name: Cosine Precision@10
179
+ - type: cosine_recall@1
180
+ value: 0.32
181
+ name: Cosine Recall@1
182
+ - type: cosine_recall@3
183
+ value: 0.56
184
+ name: Cosine Recall@3
185
+ - type: cosine_recall@5
186
+ value: 0.66
187
+ name: Cosine Recall@5
188
+ - type: cosine_recall@10
189
+ value: 0.82
190
+ name: Cosine Recall@10
191
+ - type: cosine_ndcg@10
192
+ value: 0.555381357077638
193
+ name: Cosine Ndcg@10
194
+ - type: cosine_mrr@10
195
+ value: 0.47249206349206346
196
+ name: Cosine Mrr@10
197
+ - type: cosine_map@100
198
+ value: 0.4797949229011178
199
+ name: Cosine Map@100
200
+ - task:
201
+ type: nano-beir
202
+ name: Nano BEIR
203
+ dataset:
204
+ name: NanoBEIR mean
205
+ type: NanoBEIR_mean
206
+ metrics:
207
+ - type: cosine_accuracy@1
208
+ value: 0.35
209
+ name: Cosine Accuracy@1
210
+ - type: cosine_accuracy@3
211
+ value: 0.6000000000000001
212
+ name: Cosine Accuracy@3
213
+ - type: cosine_accuracy@5
214
+ value: 0.6799999999999999
215
+ name: Cosine Accuracy@5
216
+ - type: cosine_accuracy@10
217
+ value: 0.81
218
+ name: Cosine Accuracy@10
219
+ - type: cosine_precision@1
220
+ value: 0.35
221
+ name: Cosine Precision@1
222
+ - type: cosine_precision@3
223
+ value: 0.2033333333333333
224
+ name: Cosine Precision@3
225
+ - type: cosine_precision@5
226
+ value: 0.138
227
+ name: Cosine Precision@5
228
+ - type: cosine_precision@10
229
+ value: 0.08199999999999999
230
+ name: Cosine Precision@10
231
+ - type: cosine_recall@1
232
+ value: 0.33999999999999997
233
+ name: Cosine Recall@1
234
+ - type: cosine_recall@3
235
+ value: 0.5900000000000001
236
+ name: Cosine Recall@3
237
+ - type: cosine_recall@5
238
+ value: 0.665
239
+ name: Cosine Recall@5
240
+ - type: cosine_recall@10
241
+ value: 0.78
242
+ name: Cosine Recall@10
243
+ - type: cosine_ndcg@10
244
+ value: 0.5613834030054919
245
+ name: Cosine Ndcg@10
246
+ - type: cosine_mrr@10
247
+ value: 0.4981190476190476
248
+ name: Cosine Mrr@10
249
+ - type: cosine_map@100
250
+ value: 0.49573675448295396
251
+ name: Cosine Map@100
252
+ ---
253
+
254
+ # SentenceTransformer based on answerdotai/ModernBERT-base
255
+
256
+ This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) on the [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
257
+
258
+ ## Model Details
259
+
260
+ ### Model Description
261
+ - **Model Type:** Sentence Transformer
262
+ - **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) <!-- at revision ba9fd55dd48a0e8a0e62b93b6ae6c7919d922a32 -->
263
+ - **Maximum Sequence Length:** 512 tokens
264
+ - **Output Dimensionality:** 768 dimensions
265
+ - **Similarity Function:** Cosine Similarity
266
+ - **Training Dataset:**
267
+ - [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq)
268
+ - **Language:** en
269
+ <!-- - **License:** Unknown -->
270
+
271
+ ### Model Sources
272
+
273
+ - **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
274
+ - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
275
+ - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
276
+
277
+ ### Full Model Architecture
278
+
279
+ ```
280
+ SentenceTransformer(
281
+ (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: ModernBertModel
282
+ (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
283
+ )
284
+ ```
285
+
286
+ ## Usage
287
+
288
+ ### Direct Usage (Sentence Transformers)
289
+
290
+ First install the Sentence Transformers library:
291
+
292
+ ```bash
293
+ pip install -U sentence-transformers
294
+ ```
295
+
296
+ Then you can load this model and run inference.
297
+ ```python
298
+ from sentence_transformers import SentenceTransformer
299
+
300
+ # Download from the 🤗 Hub
301
+ model = SentenceTransformer("tomaarsen/ModernBERT-base-gooaq-8e-05")
302
+ # Run inference
303
+ sentences = [
304
+ 'are you human korean novela?',
305
+ "Are You Human? (Korean: 너도 인간이니; RR: Neodo Inganini; lit. Are You Human Too?) is a 2018 South Korean television series starring Seo Kang-jun and Gong Seung-yeon. It aired on KBS2's Mondays and Tuesdays at 22:00 (KST) time slot, from June 4 to August 7, 2018.",
306
+ 'A relative of European pear varieties like Bartlett and Anjou, the Asian pear is great used in recipes or simply eaten out of hand. It retains a crispness that works well in slaws and salads, and it holds its shape better than European pears when baked and cooked.',
307
+ ]
308
+ embeddings = model.encode(sentences)
309
+ print(embeddings.shape)
310
+ # [3, 768]
311
+
312
+ # Get the similarity scores for the embeddings
313
+ similarities = model.similarity(embeddings, embeddings)
314
+ print(similarities.shape)
315
+ # [3, 3]
316
+ ```
317
+
318
+ <!--
319
+ ### Direct Usage (Transformers)
320
+
321
+ <details><summary>Click to see the direct usage in Transformers</summary>
322
+
323
+ </details>
324
+ -->
325
+
326
+ <!--
327
+ ### Downstream Usage (Sentence Transformers)
328
+
329
+ You can finetune this model on your own dataset.
330
+
331
+ <details><summary>Click to expand</summary>
332
+
333
+ </details>
334
+ -->
335
+
336
+ <!--
337
+ ### Out-of-Scope Use
338
+
339
+ *List how the model may foreseeably be misused and address what users ought not to do with the model.*
340
+ -->
341
+
342
+ ## Evaluation
343
+
344
+ ### Metrics
345
+
346
+ #### Information Retrieval
347
+
348
+ * Datasets: `NanoNQ` and `NanoMSMARCO`
349
+ * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator)
350
+
351
+ | Metric | NanoNQ | NanoMSMARCO |
352
+ |:--------------------|:-----------|:------------|
353
+ | cosine_accuracy@1 | 0.38 | 0.32 |
354
+ | cosine_accuracy@3 | 0.64 | 0.56 |
355
+ | cosine_accuracy@5 | 0.7 | 0.66 |
356
+ | cosine_accuracy@10 | 0.8 | 0.82 |
357
+ | cosine_precision@1 | 0.38 | 0.32 |
358
+ | cosine_precision@3 | 0.22 | 0.1867 |
359
+ | cosine_precision@5 | 0.144 | 0.132 |
360
+ | cosine_precision@10 | 0.082 | 0.082 |
361
+ | cosine_recall@1 | 0.36 | 0.32 |
362
+ | cosine_recall@3 | 0.62 | 0.56 |
363
+ | cosine_recall@5 | 0.67 | 0.66 |
364
+ | cosine_recall@10 | 0.74 | 0.82 |
365
+ | **cosine_ndcg@10** | **0.5674** | **0.5554** |
366
+ | cosine_mrr@10 | 0.5237 | 0.4725 |
367
+ | cosine_map@100 | 0.5117 | 0.4798 |
368
+
369
+ #### Nano BEIR
370
+
371
+ * Dataset: `NanoBEIR_mean`
372
+ * Evaluated with [<code>NanoBEIREvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.NanoBEIREvaluator)
373
+
374
+ | Metric | Value |
375
+ |:--------------------|:-----------|
376
+ | cosine_accuracy@1 | 0.35 |
377
+ | cosine_accuracy@3 | 0.6 |
378
+ | cosine_accuracy@5 | 0.68 |
379
+ | cosine_accuracy@10 | 0.81 |
380
+ | cosine_precision@1 | 0.35 |
381
+ | cosine_precision@3 | 0.2033 |
382
+ | cosine_precision@5 | 0.138 |
383
+ | cosine_precision@10 | 0.082 |
384
+ | cosine_recall@1 | 0.34 |
385
+ | cosine_recall@3 | 0.59 |
386
+ | cosine_recall@5 | 0.665 |
387
+ | cosine_recall@10 | 0.78 |
388
+ | **cosine_ndcg@10** | **0.5614** |
389
+ | cosine_mrr@10 | 0.4981 |
390
+ | cosine_map@100 | 0.4957 |
391
+
392
+ <!--
393
+ ## Bias, Risks and Limitations
394
+
395
+ *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
396
+ -->
397
+
398
+ <!--
399
+ ### Recommendations
400
+
401
+ *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
402
+ -->
403
+
404
+ ## Training Details
405
+
406
+ ### Training Dataset
407
+
408
+ #### gooaq
409
+
410
+ * Dataset: [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) at [b089f72](https://huggingface.co/datasets/sentence-transformers/gooaq/tree/b089f728748a068b7bc5234e5bcf5b25e3c8279c)
411
+ * Size: 3,012,496 training samples
412
+ * Columns: <code>question</code> and <code>answer</code>
413
+ * Approximate statistics based on the first 1000 samples:
414
+ | | question | answer |
415
+ |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
416
+ | type | string | string |
417
+ | details | <ul><li>min: 8 tokens</li><li>mean: 12.0 tokens</li><li>max: 21 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 58.17 tokens</li><li>max: 190 tokens</li></ul> |
418
+ * Samples:
419
+ | question | answer |
420
+ |:-----------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
421
+ | <code>what is the difference between clay and mud mask?</code> | <code>The main difference between the two is that mud is a skin-healing agent, while clay is a cosmetic, drying agent. Clay masks are most useful for someone who has oily skin and is prone to breakouts of acne and blemishes.</code> |
422
+ | <code>myki how much on card?</code> | <code>A full fare myki card costs $6 and a concession, seniors or child myki costs $3. For more information about how to use your myki, visit ptv.vic.gov.au or call 1800 800 007.</code> |
423
+ | <code>how to find out if someone blocked your phone number on iphone?</code> | <code>If you get a notification like "Message Not Delivered" or you get no notification at all, that's a sign of a potential block. Next, you could try calling the person. If the call goes right to voicemail or rings once (or a half ring) then goes to voicemail, that's further evidence you may have been blocked.</code> |
424
+ * Loss: [<code>CachedMultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters:
425
+ ```json
426
+ {
427
+ "scale": 20.0,
428
+ "similarity_fct": "cos_sim"
429
+ }
430
+ ```
431
+
432
+ ### Evaluation Dataset
433
+
434
+ #### gooaq
435
+
436
+ * Dataset: [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) at [b089f72](https://huggingface.co/datasets/sentence-transformers/gooaq/tree/b089f728748a068b7bc5234e5bcf5b25e3c8279c)
437
+ * Size: 3,012,496 evaluation samples
438
+ * Columns: <code>question</code> and <code>answer</code>
439
+ * Approximate statistics based on the first 1000 samples:
440
+ | | question | answer |
441
+ |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
442
+ | type | string | string |
443
+ | details | <ul><li>min: 8 tokens</li><li>mean: 12.05 tokens</li><li>max: 21 tokens</li></ul> | <ul><li>min: 13 tokens</li><li>mean: 59.08 tokens</li><li>max: 116 tokens</li></ul> |
444
+ * Samples:
445
+ | question | answer |
446
+ |:-----------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
447
+ | <code>how do i program my directv remote with my tv?</code> | <code>['Press MENU on your remote.', 'Select Settings & Help > Settings > Remote Control > Program Remote.', 'Choose the device (TV, audio, DVD) you wish to program. ... ', 'Follow the on-screen prompts to complete programming.']</code> |
448
+ | <code>are rodrigues fruit bats nocturnal?</code> | <code>Before its numbers were threatened by habitat destruction, storms, and hunting, some of those groups could number 500 or more members. Sunrise, sunset. Rodrigues fruit bats are most active at dawn, at dusk, and at night.</code> |
449
+ | <code>why does your heart rate increase during exercise bbc bitesize?</code> | <code>During exercise there is an increase in physical activity and muscle cells respire more than they do when the body is at rest. The heart rate increases during exercise. The rate and depth of breathing increases - this makes sure that more oxygen is absorbed into the blood, and more carbon dioxide is removed from it.</code> |
450
+ * Loss: [<code>CachedMultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters:
451
+ ```json
452
+ {
453
+ "scale": 20.0,
454
+ "similarity_fct": "cos_sim"
455
+ }
456
+ ```
457
+
458
+ ### Training Hyperparameters
459
+ #### Non-Default Hyperparameters
460
+
461
+ - `eval_strategy`: steps
462
+ - `per_device_train_batch_size`: 2048
463
+ - `per_device_eval_batch_size`: 2048
464
+ - `learning_rate`: 8e-05
465
+ - `num_train_epochs`: 1
466
+ - `warmup_ratio`: 0.05
467
+ - `bf16`: True
468
+ - `batch_sampler`: no_duplicates
469
+
470
+ #### All Hyperparameters
471
+ <details><summary>Click to expand</summary>
472
+
473
+ - `overwrite_output_dir`: False
474
+ - `do_predict`: False
475
+ - `eval_strategy`: steps
476
+ - `prediction_loss_only`: True
477
+ - `per_device_train_batch_size`: 2048
478
+ - `per_device_eval_batch_size`: 2048
479
+ - `per_gpu_train_batch_size`: None
480
+ - `per_gpu_eval_batch_size`: None
481
+ - `gradient_accumulation_steps`: 1
482
+ - `eval_accumulation_steps`: None
483
+ - `torch_empty_cache_steps`: None
484
+ - `learning_rate`: 8e-05
485
+ - `weight_decay`: 0.0
486
+ - `adam_beta1`: 0.9
487
+ - `adam_beta2`: 0.999
488
+ - `adam_epsilon`: 1e-08
489
+ - `max_grad_norm`: 1.0
490
+ - `num_train_epochs`: 1
491
+ - `max_steps`: -1
492
+ - `lr_scheduler_type`: linear
493
+ - `lr_scheduler_kwargs`: {}
494
+ - `warmup_ratio`: 0.05
495
+ - `warmup_steps`: 0
496
+ - `log_level`: passive
497
+ - `log_level_replica`: warning
498
+ - `log_on_each_node`: True
499
+ - `logging_nan_inf_filter`: True
500
+ - `save_safetensors`: True
501
+ - `save_on_each_node`: False
502
+ - `save_only_model`: False
503
+ - `restore_callback_states_from_checkpoint`: False
504
+ - `no_cuda`: False
505
+ - `use_cpu`: False
506
+ - `use_mps_device`: False
507
+ - `seed`: 42
508
+ - `data_seed`: None
509
+ - `jit_mode_eval`: False
510
+ - `use_ipex`: False
511
+ - `bf16`: True
512
+ - `fp16`: False
513
+ - `fp16_opt_level`: O1
514
+ - `half_precision_backend`: auto
515
+ - `bf16_full_eval`: False
516
+ - `fp16_full_eval`: False
517
+ - `tf32`: None
518
+ - `local_rank`: 0
519
+ - `ddp_backend`: None
520
+ - `tpu_num_cores`: None
521
+ - `tpu_metrics_debug`: False
522
+ - `debug`: []
523
+ - `dataloader_drop_last`: False
524
+ - `dataloader_num_workers`: 0
525
+ - `dataloader_prefetch_factor`: None
526
+ - `past_index`: -1
527
+ - `disable_tqdm`: False
528
+ - `remove_unused_columns`: True
529
+ - `label_names`: None
530
+ - `load_best_model_at_end`: False
531
+ - `ignore_data_skip`: False
532
+ - `fsdp`: []
533
+ - `fsdp_min_num_params`: 0
534
+ - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
535
+ - `fsdp_transformer_layer_cls_to_wrap`: None
536
+ - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
537
+ - `deepspeed`: None
538
+ - `label_smoothing_factor`: 0.0
539
+ - `optim`: adamw_torch
540
+ - `optim_args`: None
541
+ - `adafactor`: False
542
+ - `group_by_length`: False
543
+ - `length_column_name`: length
544
+ - `ddp_find_unused_parameters`: None
545
+ - `ddp_bucket_cap_mb`: None
546
+ - `ddp_broadcast_buffers`: False
547
+ - `dataloader_pin_memory`: True
548
+ - `dataloader_persistent_workers`: False
549
+ - `skip_memory_metrics`: True
550
+ - `use_legacy_prediction_loop`: False
551
+ - `push_to_hub`: False
552
+ - `resume_from_checkpoint`: None
553
+ - `hub_model_id`: None
554
+ - `hub_strategy`: every_save
555
+ - `hub_private_repo`: None
556
+ - `hub_always_push`: False
557
+ - `gradient_checkpointing`: False
558
+ - `gradient_checkpointing_kwargs`: None
559
+ - `include_inputs_for_metrics`: False
560
+ - `include_for_metrics`: []
561
+ - `eval_do_concat_batches`: True
562
+ - `fp16_backend`: auto
563
+ - `push_to_hub_model_id`: None
564
+ - `push_to_hub_organization`: None
565
+ - `mp_parameters`:
566
+ - `auto_find_batch_size`: False
567
+ - `full_determinism`: False
568
+ - `torchdynamo`: None
569
+ - `ray_scope`: last
570
+ - `ddp_timeout`: 1800
571
+ - `torch_compile`: False
572
+ - `torch_compile_backend`: None
573
+ - `torch_compile_mode`: None
574
+ - `dispatch_batches`: None
575
+ - `split_batches`: None
576
+ - `include_tokens_per_second`: False
577
+ - `include_num_input_tokens_seen`: False
578
+ - `neftune_noise_alpha`: None
579
+ - `optim_target_modules`: None
580
+ - `batch_eval_metrics`: False
581
+ - `eval_on_start`: False
582
+ - `use_liger_kernel`: False
583
+ - `eval_use_gather_object`: False
584
+ - `average_tokens_across_devices`: False
585
+ - `prompts`: None
586
+ - `batch_sampler`: no_duplicates
587
+ - `multi_dataset_batch_sampler`: proportional
588
+
589
+ </details>
590
+
591
+ ### Training Logs
592
+ <details><summary>Click to expand</summary>
593
+
594
+ | Epoch | Step | Training Loss | Validation Loss | NanoNQ_cosine_ndcg@10 | NanoMSMARCO_cosine_ndcg@10 | NanoBEIR_mean_cosine_ndcg@10 |
595
+ |:------:|:----:|:-------------:|:---------------:|:---------------------:|:--------------------------:|:----------------------------:|
596
+ | 0 | 0 | - | - | 0.0388 | 0.0785 | 0.0587 |
597
+ | 0.0068 | 10 | 6.9066 | - | - | - | - |
598
+ | 0.0136 | 20 | 4.853 | - | - | - | - |
599
+ | 0.0204 | 30 | 2.5305 | - | - | - | - |
600
+ | 0.0272 | 40 | 1.3877 | - | - | - | - |
601
+ | 0.0340 | 50 | 0.871 | 0.3358 | 0.4385 | 0.4897 | 0.4641 |
602
+ | 0.0408 | 60 | 0.6463 | - | - | - | - |
603
+ | 0.0476 | 70 | 0.5336 | - | - | - | - |
604
+ | 0.0544 | 80 | 0.4601 | - | - | - | - |
605
+ | 0.0612 | 90 | 0.4057 | - | - | - | - |
606
+ | 0.0680 | 100 | 0.366 | 0.1523 | 0.5100 | 0.4477 | 0.4789 |
607
+ | 0.0748 | 110 | 0.3498 | - | - | - | - |
608
+ | 0.0816 | 120 | 0.3297 | - | - | - | - |
609
+ | 0.0884 | 130 | 0.3038 | - | - | - | - |
610
+ | 0.0952 | 140 | 0.3062 | - | - | - | - |
611
+ | 0.1020 | 150 | 0.2976 | 0.1176 | 0.5550 | 0.4742 | 0.5146 |
612
+ | 0.1088 | 160 | 0.2843 | - | - | - | - |
613
+ | 0.1156 | 170 | 0.2732 | - | - | - | - |
614
+ | 0.1224 | 180 | 0.2549 | - | - | - | - |
615
+ | 0.1292 | 190 | 0.2584 | - | - | - | - |
616
+ | 0.1360 | 200 | 0.2451 | 0.1018 | 0.5313 | 0.4846 | 0.5079 |
617
+ | 0.1428 | 210 | 0.2521 | - | - | - | - |
618
+ | 0.1496 | 220 | 0.2451 | - | - | - | - |
619
+ | 0.1564 | 230 | 0.2367 | - | - | - | - |
620
+ | 0.1632 | 240 | 0.2359 | - | - | - | - |
621
+ | 0.1700 | 250 | 0.2343 | 0.0947 | 0.5489 | 0.4823 | 0.5156 |
622
+ | 0.1768 | 260 | 0.2263 | - | - | - | - |
623
+ | 0.1835 | 270 | 0.2225 | - | - | - | - |
624
+ | 0.1903 | 280 | 0.2219 | - | - | - | - |
625
+ | 0.1971 | 290 | 0.2136 | - | - | - | - |
626
+ | 0.2039 | 300 | 0.2202 | 0.0932 | 0.5165 | 0.4674 | 0.4920 |
627
+ | 0.2107 | 310 | 0.2198 | - | - | - | - |
628
+ | 0.2175 | 320 | 0.21 | - | - | - | - |
629
+ | 0.2243 | 330 | 0.207 | - | - | - | - |
630
+ | 0.2311 | 340 | 0.1972 | - | - | - | - |
631
+ | 0.2379 | 350 | 0.2037 | 0.0877 | 0.5231 | 0.5039 | 0.5135 |
632
+ | 0.2447 | 360 | 0.2054 | - | - | - | - |
633
+ | 0.2515 | 370 | 0.197 | - | - | - | - |
634
+ | 0.2583 | 380 | 0.1922 | - | - | - | - |
635
+ | 0.2651 | 390 | 0.1965 | - | - | - | - |
636
+ | 0.2719 | 400 | 0.1962 | 0.0843 | 0.5409 | 0.4746 | 0.5078 |
637
+ | 0.2787 | 410 | 0.186 | - | - | - | - |
638
+ | 0.2855 | 420 | 0.1911 | - | - | - | - |
639
+ | 0.2923 | 430 | 0.1969 | - | - | - | - |
640
+ | 0.2991 | 440 | 0.193 | - | - | - | - |
641
+ | 0.3059 | 450 | 0.1912 | 0.0763 | 0.5398 | 0.5083 | 0.5241 |
642
+ | 0.3127 | 460 | 0.1819 | - | - | - | - |
643
+ | 0.3195 | 470 | 0.1873 | - | - | - | - |
644
+ | 0.3263 | 480 | 0.1899 | - | - | - | - |
645
+ | 0.3331 | 490 | 0.1764 | - | - | - | - |
646
+ | 0.3399 | 500 | 0.1828 | 0.0728 | 0.5439 | 0.5176 | 0.5308 |
647
+ | 0.3467 | 510 | 0.1753 | - | - | - | - |
648
+ | 0.3535 | 520 | 0.1725 | - | - | - | - |
649
+ | 0.3603 | 530 | 0.1758 | - | - | - | - |
650
+ | 0.3671 | 540 | 0.183 | - | - | - | - |
651
+ | 0.3739 | 550 | 0.1789 | 0.0733 | 0.5437 | 0.5185 | 0.5311 |
652
+ | 0.3807 | 560 | 0.1773 | - | - | - | - |
653
+ | 0.3875 | 570 | 0.1764 | - | - | - | - |
654
+ | 0.3943 | 580 | 0.1638 | - | - | - | - |
655
+ | 0.4011 | 590 | 0.1809 | - | - | - | - |
656
+ | 0.4079 | 600 | 0.1727 | 0.0700 | 0.5550 | 0.5021 | 0.5286 |
657
+ | 0.4147 | 610 | 0.1664 | - | - | - | - |
658
+ | 0.4215 | 620 | 0.1683 | - | - | - | - |
659
+ | 0.4283 | 630 | 0.1622 | - | - | - | - |
660
+ | 0.4351 | 640 | 0.1592 | - | - | - | - |
661
+ | 0.4419 | 650 | 0.168 | 0.0662 | 0.5576 | 0.4843 | 0.5210 |
662
+ | 0.4487 | 660 | 0.1696 | - | - | - | - |
663
+ | 0.4555 | 670 | 0.1609 | - | - | - | - |
664
+ | 0.4623 | 680 | 0.1644 | - | - | - | - |
665
+ | 0.4691 | 690 | 0.1643 | - | - | - | - |
666
+ | 0.4759 | 700 | 0.1604 | 0.0660 | 0.5605 | 0.5042 | 0.5323 |
667
+ | 0.4827 | 710 | 0.1634 | - | - | - | - |
668
+ | 0.4895 | 720 | 0.1515 | - | - | - | - |
669
+ | 0.4963 | 730 | 0.1592 | - | - | - | - |
670
+ | 0.5031 | 740 | 0.1597 | - | - | - | - |
671
+ | 0.5099 | 750 | 0.1617 | 0.0643 | 0.5576 | 0.4830 | 0.5203 |
672
+ | 0.5167 | 760 | 0.1512 | - | - | - | - |
673
+ | 0.5235 | 770 | 0.1563 | - | - | - | - |
674
+ | 0.5303 | 780 | 0.1529 | - | - | - | - |
675
+ | 0.5370 | 790 | 0.1547 | - | - | - | - |
676
+ | 0.5438 | 800 | 0.1548 | 0.0620 | 0.5538 | 0.5271 | 0.5405 |
677
+ | 0.5506 | 810 | 0.1533 | - | - | - | - |
678
+ | 0.5574 | 820 | 0.1504 | - | - | - | - |
679
+ | 0.5642 | 830 | 0.1489 | - | - | - | - |
680
+ | 0.5710 | 840 | 0.1534 | - | - | - | - |
681
+ | 0.5778 | 850 | 0.1507 | 0.0611 | 0.5697 | 0.5095 | 0.5396 |
682
+ | 0.5846 | 860 | 0.1475 | - | - | - | - |
683
+ | 0.5914 | 870 | 0.1474 | - | - | - | - |
684
+ | 0.5982 | 880 | 0.1499 | - | - | - | - |
685
+ | 0.6050 | 890 | 0.1454 | - | - | - | - |
686
+ | 0.6118 | 900 | 0.1419 | 0.0620 | 0.5586 | 0.5229 | 0.5407 |
687
+ | 0.6186 | 910 | 0.1465 | - | - | - | - |
688
+ | 0.6254 | 920 | 0.1436 | - | - | - | - |
689
+ | 0.6322 | 930 | 0.1464 | - | - | - | - |
690
+ | 0.6390 | 940 | 0.1418 | - | - | - | - |
691
+ | 0.6458 | 950 | 0.1443 | 0.0565 | 0.5627 | 0.5458 | 0.5543 |
692
+ | 0.6526 | 960 | 0.1458 | - | - | - | - |
693
+ | 0.6594 | 970 | 0.1431 | - | - | - | - |
694
+ | 0.6662 | 980 | 0.1417 | - | - | - | - |
695
+ | 0.6730 | 990 | 0.1402 | - | - | - | - |
696
+ | 0.6798 | 1000 | 0.1431 | 0.0563 | 0.5499 | 0.5366 | 0.5432 |
697
+ | 0.6866 | 1010 | 0.1386 | - | - | - | - |
698
+ | 0.6934 | 1020 | 0.1413 | - | - | - | - |
699
+ | 0.7002 | 1030 | 0.1381 | - | - | - | - |
700
+ | 0.7070 | 1040 | 0.1364 | - | - | - | - |
701
+ | 0.7138 | 1050 | 0.1346 | 0.0545 | 0.5574 | 0.5416 | 0.5495 |
702
+ | 0.7206 | 1060 | 0.1338 | - | - | - | - |
703
+ | 0.7274 | 1070 | 0.1378 | - | - | - | - |
704
+ | 0.7342 | 1080 | 0.135 | - | - | - | - |
705
+ | 0.7410 | 1090 | 0.1336 | - | - | - | - |
706
+ | 0.7478 | 1100 | 0.1393 | 0.0541 | 0.5776 | 0.5362 | 0.5569 |
707
+ | 0.7546 | 1110 | 0.1427 | - | - | - | - |
708
+ | 0.7614 | 1120 | 0.1378 | - | - | - | - |
709
+ | 0.7682 | 1130 | 0.1346 | - | - | - | - |
710
+ | 0.7750 | 1140 | 0.1423 | - | - | - | - |
711
+ | 0.7818 | 1150 | 0.1368 | 0.0525 | 0.5681 | 0.5237 | 0.5459 |
712
+ | 0.7886 | 1160 | 0.1392 | - | - | - | - |
713
+ | 0.7954 | 1170 | 0.1321 | - | - | - | - |
714
+ | 0.8022 | 1180 | 0.1387 | - | - | - | - |
715
+ | 0.8090 | 1190 | 0.134 | - | - | - | - |
716
+ | 0.8158 | 1200 | 0.1369 | 0.0515 | 0.5613 | 0.5416 | 0.5514 |
717
+ | 0.8226 | 1210 | 0.1358 | - | - | - | - |
718
+ | 0.8294 | 1220 | 0.1401 | - | - | - | - |
719
+ | 0.8362 | 1230 | 0.1334 | - | - | - | - |
720
+ | 0.8430 | 1240 | 0.1331 | - | - | - | - |
721
+ | 0.8498 | 1250 | 0.1324 | 0.0510 | 0.5463 | 0.5546 | 0.5505 |
722
+ | 0.8566 | 1260 | 0.135 | - | - | - | - |
723
+ | 0.8634 | 1270 | 0.1367 | - | - | - | - |
724
+ | 0.8702 | 1280 | 0.1356 | - | - | - | - |
725
+ | 0.8770 | 1290 | 0.1291 | - | - | - | - |
726
+ | 0.8838 | 1300 | 0.1313 | 0.0498 | 0.5787 | 0.5552 | 0.5670 |
727
+ | 0.8906 | 1310 | 0.1334 | - | - | - | - |
728
+ | 0.8973 | 1320 | 0.1389 | - | - | - | - |
729
+ | 0.9041 | 1330 | 0.1302 | - | - | - | - |
730
+ | 0.9109 | 1340 | 0.1319 | - | - | - | - |
731
+ | 0.9177 | 1350 | 0.1276 | 0.0504 | 0.5757 | 0.5575 | 0.5666 |
732
+ | 0.9245 | 1360 | 0.1355 | - | - | - | - |
733
+ | 0.9313 | 1370 | 0.1289 | - | - | - | - |
734
+ | 0.9381 | 1380 | 0.1335 | - | - | - | - |
735
+ | 0.9449 | 1390 | 0.1298 | - | - | - | - |
736
+ | 0.9517 | 1400 | 0.1279 | 0.0497 | 0.5743 | 0.5567 | 0.5655 |
737
+ | 0.9585 | 1410 | 0.1324 | - | - | - | - |
738
+ | 0.9653 | 1420 | 0.1306 | - | - | - | - |
739
+ | 0.9721 | 1430 | 0.1313 | - | - | - | - |
740
+ | 0.9789 | 1440 | 0.135 | - | - | - | - |
741
+ | 0.9857 | 1450 | 0.1293 | 0.0493 | 0.5671 | 0.5554 | 0.5612 |
742
+ | 0.9925 | 1460 | 0.133 | - | - | - | - |
743
+ | 0.9993 | 1470 | 0.1213 | - | - | - | - |
744
+ | 1.0 | 1471 | - | - | 0.5674 | 0.5554 | 0.5614 |
745
+
746
+ </details>
747
+
748
+ ### Framework Versions
749
+ - Python: 3.11.10
750
+ - Sentence Transformers: 3.3.1
751
+ - Transformers: 4.48.0.dev0
752
+ - PyTorch: 2.6.0.dev20241112+cu121
753
+ - Accelerate: 1.2.0
754
+ - Datasets: 3.2.0
755
+ - Tokenizers: 0.21.0
756
+
757
+ ## Citation
758
+
759
+ ### BibTeX
760
+
761
+ #### Sentence Transformers
762
+ ```bibtex
763
+ @inproceedings{reimers-2019-sentence-bert,
764
+ title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
765
+ author = "Reimers, Nils and Gurevych, Iryna",
766
+ booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
767
+ month = "11",
768
+ year = "2019",
769
+ publisher = "Association for Computational Linguistics",
770
+ url = "https://arxiv.org/abs/1908.10084",
771
+ }
772
+ ```
773
+
774
+ #### CachedMultipleNegativesRankingLoss
775
+ ```bibtex
776
+ @misc{gao2021scaling,
777
+ title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
778
+ author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
779
+ year={2021},
780
+ eprint={2101.06983},
781
+ archivePrefix={arXiv},
782
+ primaryClass={cs.LG}
783
+ }
784
+ ```
785
+
786
+ <!--
787
+ ## Glossary
788
+
789
+ *Clearly define terms in order to be accessible across audiences.*
790
+ -->
791
+
792
+ <!--
793
+ ## Model Card Authors
794
+
795
+ *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
796
+ -->
797
+
798
+ <!--
799
+ ## Model Card Contact
800
+
801
+ *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
802
+ -->
config.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "answerdotai/ModernBERT-base",
3
+ "architectures": [
4
+ "ModernBertModel"
5
+ ],
6
+ "attention_bias": false,
7
+ "attention_dropout": 0.0,
8
+ "bos_token_id": 50281,
9
+ "classifier_activation": "gelu",
10
+ "classifier_bias": false,
11
+ "classifier_dropout": 0.0,
12
+ "classifier_pooling": "mean",
13
+ "cls_token_id": 50281,
14
+ "decoder_bias": true,
15
+ "deterministic_flash_attn": false,
16
+ "embedding_dropout": 0.0,
17
+ "eos_token_id": 50282,
18
+ "global_attn_every_n_layers": 3,
19
+ "global_rope_theta": 160000.0,
20
+ "gradient_checkpointing": false,
21
+ "hidden_activation": "gelu",
22
+ "hidden_size": 768,
23
+ "initializer_cutoff_factor": 2.0,
24
+ "initializer_range": 0.02,
25
+ "intermediate_size": 1152,
26
+ "layer_norm_eps": 1e-12,
27
+ "local_attention": 128,
28
+ "local_rope_theta": 10000.0,
29
+ "max_position_embeddings": 512,
30
+ "mlp_bias": false,
31
+ "mlp_dropout": 0.0,
32
+ "model_type": "modernbert",
33
+ "norm_bias": false,
34
+ "norm_eps": 1e-05,
35
+ "num_attention_heads": 12,
36
+ "num_hidden_layers": 22,
37
+ "pad_token_id": 50283,
38
+ "position_embedding_type": "absolute",
39
+ "reference_compile": true,
40
+ "sep_token_id": 50282,
41
+ "sparse_pred_ignore_index": -100,
42
+ "sparse_prediction": false,
43
+ "torch_dtype": "float32",
44
+ "transformers_version": "4.48.0.dev0",
45
+ "unpad_no_grad": true,
46
+ "vocab_size": 50368
47
+ }
config_sentence_transformers.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "__version__": {
3
+ "sentence_transformers": "3.3.1",
4
+ "transformers": "4.48.0.dev0",
5
+ "pytorch": "2.6.0.dev20241112+cu121"
6
+ },
7
+ "prompts": {},
8
+ "default_prompt_name": null,
9
+ "similarity_fn_name": "cosine"
10
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f81d2fd72230569932fa462520bc0d3fa21e5c9e4e1b6277670594aa0b814dc1
3
+ size 596070136
modules.json ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "idx": 0,
4
+ "name": "0",
5
+ "path": "",
6
+ "type": "sentence_transformers.models.Transformer"
7
+ },
8
+ {
9
+ "idx": 1,
10
+ "name": "1",
11
+ "path": "1_Pooling",
12
+ "type": "sentence_transformers.models.Pooling"
13
+ }
14
+ ]
sentence_bert_config.json ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {
2
+ "max_seq_length": 512,
3
+ "do_lower_case": false
4
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": {
3
+ "content": "[CLS]",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "mask_token": {
10
+ "content": "[MASK]",
11
+ "lstrip": true,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": {
17
+ "content": "[PAD]",
18
+ "lstrip": false,
19
+ "normalized": false,
20
+ "rstrip": false,
21
+ "single_word": false
22
+ },
23
+ "sep_token": {
24
+ "content": "[SEP]",
25
+ "lstrip": false,
26
+ "normalized": false,
27
+ "rstrip": false,
28
+ "single_word": false
29
+ },
30
+ "unk_token": {
31
+ "content": "[UNK]",
32
+ "lstrip": false,
33
+ "normalized": false,
34
+ "rstrip": false,
35
+ "single_word": false
36
+ }
37
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,945 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "|||IP_ADDRESS|||",
5
+ "lstrip": false,
6
+ "normalized": true,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": false
10
+ },
11
+ "1": {
12
+ "content": "<|padding|>",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "50254": {
20
+ "content": " ",
21
+ "lstrip": false,
22
+ "normalized": true,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": false
26
+ },
27
+ "50255": {
28
+ "content": " ",
29
+ "lstrip": false,
30
+ "normalized": true,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": false
34
+ },
35
+ "50256": {
36
+ "content": " ",
37
+ "lstrip": false,
38
+ "normalized": true,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": false
42
+ },
43
+ "50257": {
44
+ "content": " ",
45
+ "lstrip": false,
46
+ "normalized": true,
47
+ "rstrip": false,
48
+ "single_word": false,
49
+ "special": false
50
+ },
51
+ "50258": {
52
+ "content": " ",
53
+ "lstrip": false,
54
+ "normalized": true,
55
+ "rstrip": false,
56
+ "single_word": false,
57
+ "special": false
58
+ },
59
+ "50259": {
60
+ "content": " ",
61
+ "lstrip": false,
62
+ "normalized": true,
63
+ "rstrip": false,
64
+ "single_word": false,
65
+ "special": false
66
+ },
67
+ "50260": {
68
+ "content": " ",
69
+ "lstrip": false,
70
+ "normalized": true,
71
+ "rstrip": false,
72
+ "single_word": false,
73
+ "special": false
74
+ },
75
+ "50261": {
76
+ "content": " ",
77
+ "lstrip": false,
78
+ "normalized": true,
79
+ "rstrip": false,
80
+ "single_word": false,
81
+ "special": false
82
+ },
83
+ "50262": {
84
+ "content": " ",
85
+ "lstrip": false,
86
+ "normalized": true,
87
+ "rstrip": false,
88
+ "single_word": false,
89
+ "special": false
90
+ },
91
+ "50263": {
92
+ "content": " ",
93
+ "lstrip": false,
94
+ "normalized": true,
95
+ "rstrip": false,
96
+ "single_word": false,
97
+ "special": false
98
+ },
99
+ "50264": {
100
+ "content": " ",
101
+ "lstrip": false,
102
+ "normalized": true,
103
+ "rstrip": false,
104
+ "single_word": false,
105
+ "special": false
106
+ },
107
+ "50265": {
108
+ "content": " ",
109
+ "lstrip": false,
110
+ "normalized": true,
111
+ "rstrip": false,
112
+ "single_word": false,
113
+ "special": false
114
+ },
115
+ "50266": {
116
+ "content": " ",
117
+ "lstrip": false,
118
+ "normalized": true,
119
+ "rstrip": false,
120
+ "single_word": false,
121
+ "special": false
122
+ },
123
+ "50267": {
124
+ "content": " ",
125
+ "lstrip": false,
126
+ "normalized": true,
127
+ "rstrip": false,
128
+ "single_word": false,
129
+ "special": false
130
+ },
131
+ "50268": {
132
+ "content": " ",
133
+ "lstrip": false,
134
+ "normalized": true,
135
+ "rstrip": false,
136
+ "single_word": false,
137
+ "special": false
138
+ },
139
+ "50269": {
140
+ "content": " ",
141
+ "lstrip": false,
142
+ "normalized": true,
143
+ "rstrip": false,
144
+ "single_word": false,
145
+ "special": false
146
+ },
147
+ "50270": {
148
+ "content": " ",
149
+ "lstrip": false,
150
+ "normalized": true,
151
+ "rstrip": false,
152
+ "single_word": false,
153
+ "special": false
154
+ },
155
+ "50271": {
156
+ "content": " ",
157
+ "lstrip": false,
158
+ "normalized": true,
159
+ "rstrip": false,
160
+ "single_word": false,
161
+ "special": false
162
+ },
163
+ "50272": {
164
+ "content": " ",
165
+ "lstrip": false,
166
+ "normalized": true,
167
+ "rstrip": false,
168
+ "single_word": false,
169
+ "special": false
170
+ },
171
+ "50273": {
172
+ "content": " ",
173
+ "lstrip": false,
174
+ "normalized": true,
175
+ "rstrip": false,
176
+ "single_word": false,
177
+ "special": false
178
+ },
179
+ "50274": {
180
+ "content": " ",
181
+ "lstrip": false,
182
+ "normalized": true,
183
+ "rstrip": false,
184
+ "single_word": false,
185
+ "special": false
186
+ },
187
+ "50275": {
188
+ "content": " ",
189
+ "lstrip": false,
190
+ "normalized": true,
191
+ "rstrip": false,
192
+ "single_word": false,
193
+ "special": false
194
+ },
195
+ "50276": {
196
+ "content": " ",
197
+ "lstrip": false,
198
+ "normalized": true,
199
+ "rstrip": false,
200
+ "single_word": false,
201
+ "special": false
202
+ },
203
+ "50277": {
204
+ "content": "|||EMAIL_ADDRESS|||",
205
+ "lstrip": false,
206
+ "normalized": true,
207
+ "rstrip": false,
208
+ "single_word": false,
209
+ "special": false
210
+ },
211
+ "50278": {
212
+ "content": "|||PHONE_NUMBER|||",
213
+ "lstrip": false,
214
+ "normalized": true,
215
+ "rstrip": false,
216
+ "single_word": false,
217
+ "special": false
218
+ },
219
+ "50279": {
220
+ "content": "<|endoftext|>",
221
+ "lstrip": false,
222
+ "normalized": false,
223
+ "rstrip": false,
224
+ "single_word": false,
225
+ "special": true
226
+ },
227
+ "50280": {
228
+ "content": "[UNK]",
229
+ "lstrip": false,
230
+ "normalized": false,
231
+ "rstrip": false,
232
+ "single_word": false,
233
+ "special": true
234
+ },
235
+ "50281": {
236
+ "content": "[CLS]",
237
+ "lstrip": false,
238
+ "normalized": false,
239
+ "rstrip": false,
240
+ "single_word": false,
241
+ "special": true
242
+ },
243
+ "50282": {
244
+ "content": "[SEP]",
245
+ "lstrip": false,
246
+ "normalized": false,
247
+ "rstrip": false,
248
+ "single_word": false,
249
+ "special": true
250
+ },
251
+ "50283": {
252
+ "content": "[PAD]",
253
+ "lstrip": false,
254
+ "normalized": false,
255
+ "rstrip": false,
256
+ "single_word": false,
257
+ "special": true
258
+ },
259
+ "50284": {
260
+ "content": "[MASK]",
261
+ "lstrip": true,
262
+ "normalized": false,
263
+ "rstrip": false,
264
+ "single_word": false,
265
+ "special": true
266
+ },
267
+ "50285": {
268
+ "content": "[unused0]",
269
+ "lstrip": false,
270
+ "normalized": true,
271
+ "rstrip": false,
272
+ "single_word": false,
273
+ "special": false
274
+ },
275
+ "50286": {
276
+ "content": "[unused1]",
277
+ "lstrip": false,
278
+ "normalized": true,
279
+ "rstrip": false,
280
+ "single_word": false,
281
+ "special": false
282
+ },
283
+ "50287": {
284
+ "content": "[unused2]",
285
+ "lstrip": false,
286
+ "normalized": true,
287
+ "rstrip": false,
288
+ "single_word": false,
289
+ "special": false
290
+ },
291
+ "50288": {
292
+ "content": "[unused3]",
293
+ "lstrip": false,
294
+ "normalized": true,
295
+ "rstrip": false,
296
+ "single_word": false,
297
+ "special": false
298
+ },
299
+ "50289": {
300
+ "content": "[unused4]",
301
+ "lstrip": false,
302
+ "normalized": true,
303
+ "rstrip": false,
304
+ "single_word": false,
305
+ "special": false
306
+ },
307
+ "50290": {
308
+ "content": "[unused5]",
309
+ "lstrip": false,
310
+ "normalized": true,
311
+ "rstrip": false,
312
+ "single_word": false,
313
+ "special": false
314
+ },
315
+ "50291": {
316
+ "content": "[unused6]",
317
+ "lstrip": false,
318
+ "normalized": true,
319
+ "rstrip": false,
320
+ "single_word": false,
321
+ "special": false
322
+ },
323
+ "50292": {
324
+ "content": "[unused7]",
325
+ "lstrip": false,
326
+ "normalized": true,
327
+ "rstrip": false,
328
+ "single_word": false,
329
+ "special": false
330
+ },
331
+ "50293": {
332
+ "content": "[unused8]",
333
+ "lstrip": false,
334
+ "normalized": true,
335
+ "rstrip": false,
336
+ "single_word": false,
337
+ "special": false
338
+ },
339
+ "50294": {
340
+ "content": "[unused9]",
341
+ "lstrip": false,
342
+ "normalized": true,
343
+ "rstrip": false,
344
+ "single_word": false,
345
+ "special": false
346
+ },
347
+ "50295": {
348
+ "content": "[unused10]",
349
+ "lstrip": false,
350
+ "normalized": true,
351
+ "rstrip": false,
352
+ "single_word": false,
353
+ "special": false
354
+ },
355
+ "50296": {
356
+ "content": "[unused11]",
357
+ "lstrip": false,
358
+ "normalized": true,
359
+ "rstrip": false,
360
+ "single_word": false,
361
+ "special": false
362
+ },
363
+ "50297": {
364
+ "content": "[unused12]",
365
+ "lstrip": false,
366
+ "normalized": true,
367
+ "rstrip": false,
368
+ "single_word": false,
369
+ "special": false
370
+ },
371
+ "50298": {
372
+ "content": "[unused13]",
373
+ "lstrip": false,
374
+ "normalized": true,
375
+ "rstrip": false,
376
+ "single_word": false,
377
+ "special": false
378
+ },
379
+ "50299": {
380
+ "content": "[unused14]",
381
+ "lstrip": false,
382
+ "normalized": true,
383
+ "rstrip": false,
384
+ "single_word": false,
385
+ "special": false
386
+ },
387
+ "50300": {
388
+ "content": "[unused15]",
389
+ "lstrip": false,
390
+ "normalized": true,
391
+ "rstrip": false,
392
+ "single_word": false,
393
+ "special": false
394
+ },
395
+ "50301": {
396
+ "content": "[unused16]",
397
+ "lstrip": false,
398
+ "normalized": true,
399
+ "rstrip": false,
400
+ "single_word": false,
401
+ "special": false
402
+ },
403
+ "50302": {
404
+ "content": "[unused17]",
405
+ "lstrip": false,
406
+ "normalized": true,
407
+ "rstrip": false,
408
+ "single_word": false,
409
+ "special": false
410
+ },
411
+ "50303": {
412
+ "content": "[unused18]",
413
+ "lstrip": false,
414
+ "normalized": true,
415
+ "rstrip": false,
416
+ "single_word": false,
417
+ "special": false
418
+ },
419
+ "50304": {
420
+ "content": "[unused19]",
421
+ "lstrip": false,
422
+ "normalized": true,
423
+ "rstrip": false,
424
+ "single_word": false,
425
+ "special": false
426
+ },
427
+ "50305": {
428
+ "content": "[unused20]",
429
+ "lstrip": false,
430
+ "normalized": true,
431
+ "rstrip": false,
432
+ "single_word": false,
433
+ "special": false
434
+ },
435
+ "50306": {
436
+ "content": "[unused21]",
437
+ "lstrip": false,
438
+ "normalized": true,
439
+ "rstrip": false,
440
+ "single_word": false,
441
+ "special": false
442
+ },
443
+ "50307": {
444
+ "content": "[unused22]",
445
+ "lstrip": false,
446
+ "normalized": true,
447
+ "rstrip": false,
448
+ "single_word": false,
449
+ "special": false
450
+ },
451
+ "50308": {
452
+ "content": "[unused23]",
453
+ "lstrip": false,
454
+ "normalized": true,
455
+ "rstrip": false,
456
+ "single_word": false,
457
+ "special": false
458
+ },
459
+ "50309": {
460
+ "content": "[unused24]",
461
+ "lstrip": false,
462
+ "normalized": true,
463
+ "rstrip": false,
464
+ "single_word": false,
465
+ "special": false
466
+ },
467
+ "50310": {
468
+ "content": "[unused25]",
469
+ "lstrip": false,
470
+ "normalized": true,
471
+ "rstrip": false,
472
+ "single_word": false,
473
+ "special": false
474
+ },
475
+ "50311": {
476
+ "content": "[unused26]",
477
+ "lstrip": false,
478
+ "normalized": true,
479
+ "rstrip": false,
480
+ "single_word": false,
481
+ "special": false
482
+ },
483
+ "50312": {
484
+ "content": "[unused27]",
485
+ "lstrip": false,
486
+ "normalized": true,
487
+ "rstrip": false,
488
+ "single_word": false,
489
+ "special": false
490
+ },
491
+ "50313": {
492
+ "content": "[unused28]",
493
+ "lstrip": false,
494
+ "normalized": true,
495
+ "rstrip": false,
496
+ "single_word": false,
497
+ "special": false
498
+ },
499
+ "50314": {
500
+ "content": "[unused29]",
501
+ "lstrip": false,
502
+ "normalized": true,
503
+ "rstrip": false,
504
+ "single_word": false,
505
+ "special": false
506
+ },
507
+ "50315": {
508
+ "content": "[unused30]",
509
+ "lstrip": false,
510
+ "normalized": true,
511
+ "rstrip": false,
512
+ "single_word": false,
513
+ "special": false
514
+ },
515
+ "50316": {
516
+ "content": "[unused31]",
517
+ "lstrip": false,
518
+ "normalized": true,
519
+ "rstrip": false,
520
+ "single_word": false,
521
+ "special": false
522
+ },
523
+ "50317": {
524
+ "content": "[unused32]",
525
+ "lstrip": false,
526
+ "normalized": true,
527
+ "rstrip": false,
528
+ "single_word": false,
529
+ "special": false
530
+ },
531
+ "50318": {
532
+ "content": "[unused33]",
533
+ "lstrip": false,
534
+ "normalized": true,
535
+ "rstrip": false,
536
+ "single_word": false,
537
+ "special": false
538
+ },
539
+ "50319": {
540
+ "content": "[unused34]",
541
+ "lstrip": false,
542
+ "normalized": true,
543
+ "rstrip": false,
544
+ "single_word": false,
545
+ "special": false
546
+ },
547
+ "50320": {
548
+ "content": "[unused35]",
549
+ "lstrip": false,
550
+ "normalized": true,
551
+ "rstrip": false,
552
+ "single_word": false,
553
+ "special": false
554
+ },
555
+ "50321": {
556
+ "content": "[unused36]",
557
+ "lstrip": false,
558
+ "normalized": true,
559
+ "rstrip": false,
560
+ "single_word": false,
561
+ "special": false
562
+ },
563
+ "50322": {
564
+ "content": "[unused37]",
565
+ "lstrip": false,
566
+ "normalized": true,
567
+ "rstrip": false,
568
+ "single_word": false,
569
+ "special": false
570
+ },
571
+ "50323": {
572
+ "content": "[unused38]",
573
+ "lstrip": false,
574
+ "normalized": true,
575
+ "rstrip": false,
576
+ "single_word": false,
577
+ "special": false
578
+ },
579
+ "50324": {
580
+ "content": "[unused39]",
581
+ "lstrip": false,
582
+ "normalized": true,
583
+ "rstrip": false,
584
+ "single_word": false,
585
+ "special": false
586
+ },
587
+ "50325": {
588
+ "content": "[unused40]",
589
+ "lstrip": false,
590
+ "normalized": true,
591
+ "rstrip": false,
592
+ "single_word": false,
593
+ "special": false
594
+ },
595
+ "50326": {
596
+ "content": "[unused41]",
597
+ "lstrip": false,
598
+ "normalized": true,
599
+ "rstrip": false,
600
+ "single_word": false,
601
+ "special": false
602
+ },
603
+ "50327": {
604
+ "content": "[unused42]",
605
+ "lstrip": false,
606
+ "normalized": true,
607
+ "rstrip": false,
608
+ "single_word": false,
609
+ "special": false
610
+ },
611
+ "50328": {
612
+ "content": "[unused43]",
613
+ "lstrip": false,
614
+ "normalized": true,
615
+ "rstrip": false,
616
+ "single_word": false,
617
+ "special": false
618
+ },
619
+ "50329": {
620
+ "content": "[unused44]",
621
+ "lstrip": false,
622
+ "normalized": true,
623
+ "rstrip": false,
624
+ "single_word": false,
625
+ "special": false
626
+ },
627
+ "50330": {
628
+ "content": "[unused45]",
629
+ "lstrip": false,
630
+ "normalized": true,
631
+ "rstrip": false,
632
+ "single_word": false,
633
+ "special": false
634
+ },
635
+ "50331": {
636
+ "content": "[unused46]",
637
+ "lstrip": false,
638
+ "normalized": true,
639
+ "rstrip": false,
640
+ "single_word": false,
641
+ "special": false
642
+ },
643
+ "50332": {
644
+ "content": "[unused47]",
645
+ "lstrip": false,
646
+ "normalized": true,
647
+ "rstrip": false,
648
+ "single_word": false,
649
+ "special": false
650
+ },
651
+ "50333": {
652
+ "content": "[unused48]",
653
+ "lstrip": false,
654
+ "normalized": true,
655
+ "rstrip": false,
656
+ "single_word": false,
657
+ "special": false
658
+ },
659
+ "50334": {
660
+ "content": "[unused49]",
661
+ "lstrip": false,
662
+ "normalized": true,
663
+ "rstrip": false,
664
+ "single_word": false,
665
+ "special": false
666
+ },
667
+ "50335": {
668
+ "content": "[unused50]",
669
+ "lstrip": false,
670
+ "normalized": true,
671
+ "rstrip": false,
672
+ "single_word": false,
673
+ "special": false
674
+ },
675
+ "50336": {
676
+ "content": "[unused51]",
677
+ "lstrip": false,
678
+ "normalized": true,
679
+ "rstrip": false,
680
+ "single_word": false,
681
+ "special": false
682
+ },
683
+ "50337": {
684
+ "content": "[unused52]",
685
+ "lstrip": false,
686
+ "normalized": true,
687
+ "rstrip": false,
688
+ "single_word": false,
689
+ "special": false
690
+ },
691
+ "50338": {
692
+ "content": "[unused53]",
693
+ "lstrip": false,
694
+ "normalized": true,
695
+ "rstrip": false,
696
+ "single_word": false,
697
+ "special": false
698
+ },
699
+ "50339": {
700
+ "content": "[unused54]",
701
+ "lstrip": false,
702
+ "normalized": true,
703
+ "rstrip": false,
704
+ "single_word": false,
705
+ "special": false
706
+ },
707
+ "50340": {
708
+ "content": "[unused55]",
709
+ "lstrip": false,
710
+ "normalized": true,
711
+ "rstrip": false,
712
+ "single_word": false,
713
+ "special": false
714
+ },
715
+ "50341": {
716
+ "content": "[unused56]",
717
+ "lstrip": false,
718
+ "normalized": true,
719
+ "rstrip": false,
720
+ "single_word": false,
721
+ "special": false
722
+ },
723
+ "50342": {
724
+ "content": "[unused57]",
725
+ "lstrip": false,
726
+ "normalized": true,
727
+ "rstrip": false,
728
+ "single_word": false,
729
+ "special": false
730
+ },
731
+ "50343": {
732
+ "content": "[unused58]",
733
+ "lstrip": false,
734
+ "normalized": true,
735
+ "rstrip": false,
736
+ "single_word": false,
737
+ "special": false
738
+ },
739
+ "50344": {
740
+ "content": "[unused59]",
741
+ "lstrip": false,
742
+ "normalized": true,
743
+ "rstrip": false,
744
+ "single_word": false,
745
+ "special": false
746
+ },
747
+ "50345": {
748
+ "content": "[unused60]",
749
+ "lstrip": false,
750
+ "normalized": true,
751
+ "rstrip": false,
752
+ "single_word": false,
753
+ "special": false
754
+ },
755
+ "50346": {
756
+ "content": "[unused61]",
757
+ "lstrip": false,
758
+ "normalized": true,
759
+ "rstrip": false,
760
+ "single_word": false,
761
+ "special": false
762
+ },
763
+ "50347": {
764
+ "content": "[unused62]",
765
+ "lstrip": false,
766
+ "normalized": true,
767
+ "rstrip": false,
768
+ "single_word": false,
769
+ "special": false
770
+ },
771
+ "50348": {
772
+ "content": "[unused63]",
773
+ "lstrip": false,
774
+ "normalized": true,
775
+ "rstrip": false,
776
+ "single_word": false,
777
+ "special": false
778
+ },
779
+ "50349": {
780
+ "content": "[unused64]",
781
+ "lstrip": false,
782
+ "normalized": true,
783
+ "rstrip": false,
784
+ "single_word": false,
785
+ "special": false
786
+ },
787
+ "50350": {
788
+ "content": "[unused65]",
789
+ "lstrip": false,
790
+ "normalized": true,
791
+ "rstrip": false,
792
+ "single_word": false,
793
+ "special": false
794
+ },
795
+ "50351": {
796
+ "content": "[unused66]",
797
+ "lstrip": false,
798
+ "normalized": true,
799
+ "rstrip": false,
800
+ "single_word": false,
801
+ "special": false
802
+ },
803
+ "50352": {
804
+ "content": "[unused67]",
805
+ "lstrip": false,
806
+ "normalized": true,
807
+ "rstrip": false,
808
+ "single_word": false,
809
+ "special": false
810
+ },
811
+ "50353": {
812
+ "content": "[unused68]",
813
+ "lstrip": false,
814
+ "normalized": true,
815
+ "rstrip": false,
816
+ "single_word": false,
817
+ "special": false
818
+ },
819
+ "50354": {
820
+ "content": "[unused69]",
821
+ "lstrip": false,
822
+ "normalized": true,
823
+ "rstrip": false,
824
+ "single_word": false,
825
+ "special": false
826
+ },
827
+ "50355": {
828
+ "content": "[unused70]",
829
+ "lstrip": false,
830
+ "normalized": true,
831
+ "rstrip": false,
832
+ "single_word": false,
833
+ "special": false
834
+ },
835
+ "50356": {
836
+ "content": "[unused71]",
837
+ "lstrip": false,
838
+ "normalized": true,
839
+ "rstrip": false,
840
+ "single_word": false,
841
+ "special": false
842
+ },
843
+ "50357": {
844
+ "content": "[unused72]",
845
+ "lstrip": false,
846
+ "normalized": true,
847
+ "rstrip": false,
848
+ "single_word": false,
849
+ "special": false
850
+ },
851
+ "50358": {
852
+ "content": "[unused73]",
853
+ "lstrip": false,
854
+ "normalized": true,
855
+ "rstrip": false,
856
+ "single_word": false,
857
+ "special": false
858
+ },
859
+ "50359": {
860
+ "content": "[unused74]",
861
+ "lstrip": false,
862
+ "normalized": true,
863
+ "rstrip": false,
864
+ "single_word": false,
865
+ "special": false
866
+ },
867
+ "50360": {
868
+ "content": "[unused75]",
869
+ "lstrip": false,
870
+ "normalized": true,
871
+ "rstrip": false,
872
+ "single_word": false,
873
+ "special": false
874
+ },
875
+ "50361": {
876
+ "content": "[unused76]",
877
+ "lstrip": false,
878
+ "normalized": true,
879
+ "rstrip": false,
880
+ "single_word": false,
881
+ "special": false
882
+ },
883
+ "50362": {
884
+ "content": "[unused77]",
885
+ "lstrip": false,
886
+ "normalized": true,
887
+ "rstrip": false,
888
+ "single_word": false,
889
+ "special": false
890
+ },
891
+ "50363": {
892
+ "content": "[unused78]",
893
+ "lstrip": false,
894
+ "normalized": true,
895
+ "rstrip": false,
896
+ "single_word": false,
897
+ "special": false
898
+ },
899
+ "50364": {
900
+ "content": "[unused79]",
901
+ "lstrip": false,
902
+ "normalized": true,
903
+ "rstrip": false,
904
+ "single_word": false,
905
+ "special": false
906
+ },
907
+ "50365": {
908
+ "content": "[unused80]",
909
+ "lstrip": false,
910
+ "normalized": true,
911
+ "rstrip": false,
912
+ "single_word": false,
913
+ "special": false
914
+ },
915
+ "50366": {
916
+ "content": "[unused81]",
917
+ "lstrip": false,
918
+ "normalized": true,
919
+ "rstrip": false,
920
+ "single_word": false,
921
+ "special": false
922
+ },
923
+ "50367": {
924
+ "content": "[unused82]",
925
+ "lstrip": false,
926
+ "normalized": true,
927
+ "rstrip": false,
928
+ "single_word": false,
929
+ "special": false
930
+ }
931
+ },
932
+ "clean_up_tokenization_spaces": true,
933
+ "cls_token": "[CLS]",
934
+ "extra_special_tokens": {},
935
+ "mask_token": "[MASK]",
936
+ "model_input_names": [
937
+ "input_ids",
938
+ "attention_mask"
939
+ ],
940
+ "model_max_length": 1000000000000000019884624838656,
941
+ "pad_token": "[PAD]",
942
+ "sep_token": "[SEP]",
943
+ "tokenizer_class": "PreTrainedTokenizerFast",
944
+ "unk_token": "[UNK]"
945
+ }