tomaarsen/NeoBERT-gooaq-8e-05

SentenceTransformer based on chandar-lab/NeoBERT

This is a sentence-transformers model finetuned from chandar-lab/NeoBERT on the gooaq dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

This model has been finetuned using train_st_gooaq.py using an RTX 3090. It used the same training script as tomaarsen/ModernBERT-base-gooaq.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: chandar-lab/NeoBERT
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • gooaq
• Language: en

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NeoBERT 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/NeoBERT-gooaq-8e-05")
# Run inference
sentences = [
    'what is the difference between 18 and 20 inch tires?',
    'The only real difference is a 20" rim would be more likely to be damaged, as you pointed out. Beyond looks, there is zero benefit for the 20" rim. Also, just the availability of tires will likely be much more limited for the larger rim. ... Tire selection is better for 18" wheels than 20" wheels.',
    'So extracurricular activities are just activities that you do outside of class. The Common App says that extracurricular activities "include arts, athletics, clubs, employment, personal commitments, and other pursuits."',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Datasets: NanoNQ and NanoMSMARCO
• Evaluated with InformationRetrievalEvaluator

Metric	NanoNQ	NanoMSMARCO
cosine_accuracy@1	0.46	0.32
cosine_accuracy@3	0.64	0.58
cosine_accuracy@5	0.7	0.68
cosine_accuracy@10	0.76	0.74
cosine_precision@1	0.46	0.32
cosine_precision@3	0.22	0.1933
cosine_precision@5	0.144	0.136
cosine_precision@10	0.08	0.074
cosine_recall@1	0.43	0.32
cosine_recall@3	0.62	0.58
cosine_recall@5	0.68	0.68
cosine_recall@10	0.73	0.74
cosine_ndcg@10	0.5921	0.5415
cosine_mrr@10	0.5607	0.4768
cosine_map@100	0.5501	0.4902

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with NanoBEIREvaluator

Metric	Value
cosine_accuracy@1	0.39
cosine_accuracy@3	0.61
cosine_accuracy@5	0.69
cosine_accuracy@10	0.75
cosine_precision@1	0.39
cosine_precision@3	0.2067
cosine_precision@5	0.14
cosine_precision@10	0.077
cosine_recall@1	0.375
cosine_recall@3	0.6
cosine_recall@5	0.68
cosine_recall@10	0.735
cosine_ndcg@10	0.5668
cosine_mrr@10	0.5188
cosine_map@100	0.5202

Training Details

Training Dataset

gooaq

• Dataset: gooaq at b089f72
• Size: 3,011,496 training samples
• Columns: question and answer
• Approximate statistics based on the first 1000 samples:

  	question	answer
  type	string	string
  details	min: 8 tokens mean: 11.87 tokens max: 23 tokens	min: 14 tokens mean: 60.09 tokens max: 201 tokens

• Samples:

  question	answer
  what is the difference between clay and mud mask?	The main difference between the two is that mud is a skin-healing agent, while clay is a cosmetic, drying agent. Clay masks are most useful for someone who has oily skin and is prone to breakouts of acne and blemishes.
  myki how much on card?	A full fare myki card costs $6 and a concession, seniors or child myki costs $3. For more information about how to use your myki, visit ptv.vic.gov.au or call 1800 800 007.
  how to find out if someone blocked your phone number on iphone?	If you get a notification like "Message Not Delivered" or you get no notification at all, that's a sign of a potential block. Next, you could try calling the person. If the call goes right to voicemail or rings once (or a half ring) then goes to voicemail, that's further evidence you may have been blocked.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

gooaq

• Dataset: gooaq at b089f72
• Size: 1,000 evaluation samples
• Columns: question and answer
• Approximate statistics based on the first 1000 samples:

  	question	answer
  type	string	string
  details	min: 8 tokens mean: 11.88 tokens max: 22 tokens	min: 14 tokens mean: 61.03 tokens max: 127 tokens

• Samples:

  question	answer
  how do i program my directv remote with my tv?	['Press MENU on your remote.', 'Select Settings & Help > Settings > Remote Control > Program Remote.', 'Choose the device (TV, audio, DVD) you wish to program. ... ', 'Follow the on-screen prompts to complete programming.']
  are rodrigues fruit bats nocturnal?	Before its numbers were threatened by habitat destruction, storms, and hunting, some of those groups could number 500 or more members. Sunrise, sunset. Rodrigues fruit bats are most active at dawn, at dusk, and at night.
  why does your heart rate increase during exercise bbc bitesize?	During exercise there is an increase in physical activity and muscle cells respire more than they do when the body is at rest. The heart rate increases during exercise. The rate and depth of breathing increases - this makes sure that more oxygen is absorbed into the blood, and more carbon dioxide is removed from it.

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 2048
• per_device_eval_batch_size: 2048
• learning_rate: 8e-05
• num_train_epochs: 1
• warmup_ratio: 0.05
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 2048
• per_device_eval_batch_size: 2048
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 8e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.05
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	Validation Loss	NanoNQ_cosine_ndcg@10	NanoMSMARCO_cosine_ndcg@10	NanoBEIR_mean_cosine_ndcg@10
-1	-1	-	-	0.0428	0.1127	0.0777
0.0068	10	4.2332	-	-	-	-
0.0136	20	1.5303	-	-	-	-
0.0204	30	0.887	-	-	-	-
0.0272	40	0.6286	-	-	-	-
0.0340	50	0.5193	0.2091	0.4434	0.4454	0.4444
0.0408	60	0.4423	-	-	-	-
0.0476	70	0.3842	-	-	-	-
0.0544	80	0.3576	-	-	-	-
0.0612	90	0.3301	-	-	-	-
0.0680	100	0.3135	0.1252	0.4606	0.5150	0.4878
0.0748	110	0.302	-	-	-	-
0.0816	120	0.277	-	-	-	-
0.0884	130	0.2694	-	-	-	-
0.0952	140	0.2628	-	-	-	-
0.1020	150	0.2471	0.0949	0.5135	0.5133	0.5134
0.1088	160	0.2343	-	-	-	-
0.1156	170	0.2386	-	-	-	-
0.1224	180	0.219	-	-	-	-
0.1292	190	0.217	-	-	-	-
0.1360	200	0.2073	0.0870	0.5281	0.4824	0.5052
0.1428	210	0.2208	-	-	-	-
0.1496	220	0.2046	-	-	-	-
0.1564	230	0.2045	-	-	-	-
0.1632	240	0.1987	-	-	-	-
0.1700	250	0.1949	0.0734	0.5781	0.4976	0.5378
0.1768	260	0.1888	-	-	-	-
0.1835	270	0.187	-	-	-	-
0.1903	280	0.1834	-	-	-	-
0.1971	290	0.1747	-	-	-	-
0.2039	300	0.1805	0.0663	0.5580	0.5453	0.5516
0.2107	310	0.1738	-	-	-	-
0.2175	320	0.1707	-	-	-	-
0.2243	330	0.1758	-	-	-	-
0.2311	340	0.1762	-	-	-	-
0.2379	350	0.1649	0.0624	0.5761	0.5310	0.5535
0.2447	360	0.1682	-	-	-	-
0.2515	370	0.1629	-	-	-	-
0.2583	380	0.1595	-	-	-	-
0.2651	390	0.1571	-	-	-	-
0.2719	400	0.1617	0.0592	0.5865	0.5193	0.5529
0.2787	410	0.1521	-	-	-	-
0.2855	420	0.1518	-	-	-	-
0.2923	430	0.1583	-	-	-	-
0.2991	440	0.1516	-	-	-	-
0.3059	450	0.1473	0.0570	0.5844	0.5181	0.5512
0.3127	460	0.1491	-	-	-	-
0.3195	470	0.1487	-	-	-	-
0.3263	480	0.1457	-	-	-	-
0.3331	490	0.1463	-	-	-	-
0.3399	500	0.141	0.0571	0.5652	0.5027	0.5340
0.3467	510	0.1438	-	-	-	-
0.3535	520	0.148	-	-	-	-
0.3603	530	0.136	-	-	-	-
0.3671	540	0.1359	-	-	-	-
0.3739	550	0.1388	0.0507	0.5457	0.4660	0.5058
0.3807	560	0.1358	-	-	-	-
0.3875	570	0.1365	-	-	-	-
0.3943	580	0.1328	-	-	-	-
0.4011	590	0.1404	-	-	-	-
0.4079	600	0.1304	0.0524	0.5477	0.5259	0.5368
0.4147	610	0.1321	-	-	-	-
0.4215	620	0.1322	-	-	-	-
0.4283	630	0.1262	-	-	-	-
0.4351	640	0.1339	-	-	-	-
0.4419	650	0.1257	0.0494	0.5564	0.4920	0.5242
0.4487	660	0.1247	-	-	-	-
0.4555	670	0.1316	-	-	-	-
0.4623	680	0.124	-	-	-	-
0.4691	690	0.1247	-	-	-	-
0.4759	700	0.1212	0.0480	0.5663	0.5040	0.5351
0.4827	710	0.1194	-	-	-	-
0.4895	720	0.1224	-	-	-	-
0.4963	730	0.1225	-	-	-	-
0.5031	740	0.1209	-	-	-	-
0.5099	750	0.1197	0.0447	0.5535	0.5127	0.5331
0.5167	760	0.1196	-	-	-	-
0.5235	770	0.1129	-	-	-	-
0.5303	780	0.1223	-	-	-	-
0.5370	790	0.1159	-	-	-	-
0.5438	800	0.1178	0.0412	0.5558	0.5275	0.5416
0.5506	810	0.1186	-	-	-	-
0.5574	820	0.1153	-	-	-	-
0.5642	830	0.1178	-	-	-	-
0.5710	840	0.1155	-	-	-	-
0.5778	850	0.1152	0.0432	0.5738	0.5243	0.5490
0.5846	860	0.1101	-	-	-	-
0.5914	870	0.1057	-	-	-	-
0.5982	880	0.1141	-	-	-	-
0.6050	890	0.1172	-	-	-	-
0.6118	900	0.1146	0.0414	0.5641	0.4805	0.5223
0.6186	910	0.1094	-	-	-	-
0.6254	920	0.1116	-	-	-	-
0.6322	930	0.111	-	-	-	-
0.6390	940	0.1078	-	-	-	-
0.6458	950	0.1041	0.0424	0.5883	0.5412	0.5647
0.6526	960	0.1068	-	-	-	-
0.6594	970	0.1076	-	-	-	-
0.6662	980	0.1068	-	-	-	-
0.6730	990	0.1038	-	-	-	-
0.6798	1000	0.1017	0.0409	0.5850	0.5117	0.5483
0.6866	1010	0.1079	-	-	-	-
0.6934	1020	0.1067	-	-	-	-
0.7002	1030	0.1079	-	-	-	-
0.7070	1040	0.1039	-	-	-	-
0.7138	1050	0.1016	0.0356	0.5927	0.5344	0.5636
0.7206	1060	0.1017	-	-	-	-
0.7274	1070	0.1029	-	-	-	-
0.7342	1080	0.1038	-	-	-	-
0.7410	1090	0.0994	-	-	-	-
0.7478	1100	0.0984	0.0376	0.5618	0.5321	0.5470
0.7546	1110	0.0966	-	-	-	-
0.7614	1120	0.1024	-	-	-	-
0.7682	1130	0.099	-	-	-	-
0.7750	1140	0.1017	-	-	-	-
0.7818	1150	0.0951	0.0368	0.5832	0.5073	0.5453
0.7886	1160	0.1008	-	-	-	-
0.7954	1170	0.096	-	-	-	-
0.8022	1180	0.0962	-	-	-	-
0.8090	1190	0.1004	-	-	-	-
0.8158	1200	0.0986	0.0321	0.5895	0.5242	0.5568
0.8226	1210	0.0966	-	-	-	-
0.8294	1220	0.096	-	-	-	-
0.8362	1230	0.0962	-	-	-	-
0.8430	1240	0.0987	-	-	-	-
0.8498	1250	0.096	0.0316	0.5801	0.5434	0.5617
0.8566	1260	0.097	-	-	-	-
0.8634	1270	0.0929	-	-	-	-
0.8702	1280	0.0973	-	-	-	-
0.8770	1290	0.0973	-	-	-	-
0.8838	1300	0.0939	0.0330	0.5916	0.5478	0.5697
0.8906	1310	0.0968	-	-	-	-
0.8973	1320	0.0969	-	-	-	-
0.9041	1330	0.0931	-	-	-	-
0.9109	1340	0.0919	-	-	-	-
0.9177	1350	0.0916	0.0324	0.5908	0.5308	0.5608
0.9245	1360	0.0903	-	-	-	-
0.9313	1370	0.0957	-	-	-	-
0.9381	1380	0.0891	-	-	-	-
0.9449	1390	0.0909	-	-	-	-
0.9517	1400	0.0924	0.0318	0.5823	0.5388	0.5605
0.9585	1410	0.0932	-	-	-	-
0.9653	1420	0.0916	-	-	-	-
0.9721	1430	0.0966	-	-	-	-
0.9789	1440	0.0864	-	-	-	-
0.9857	1450	0.0872	0.0311	0.5895	0.5442	0.5668
0.9925	1460	0.0897	-	-	-	-
0.9993	1470	0.086	-	-	-	-
-1	-1	-	-	0.5921	0.5415	0.5668

Framework Versions

• Python: 3.11.10
• Sentence Transformers: 3.5.0.dev0
• Transformers: 4.49.0
• PyTorch: 2.5.1+cu124
• Accelerate: 1.2.0
• Datasets: 2.21.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}