joshuapb/fine-tuned-matryoshka

BGE base Financial Matryoshka

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("joshuapb/fine-tuned-matryoshka")
# Run inference
sentences = [
    'Verbalized number or word (e.g. “lowest”, “low”, “medium”, “high”, “highest”), such as "Confidence: 60% / Medium".\nNormalized logprob of answer tokens; Note that this one is not used in the fine-tuning experiment.\nLogprob of an indirect "True/False" token after the raw answer.\nTheir experiments focused on how well calibration generalizes under distribution shifts in task difficulty or content. Each fine-tuning datapoint is a question, the model’s answer (possibly incorrect), and a calibrated confidence. Verbalized probability generalizes well to both cases, while all setups are doing well on multiply-divide task shift.  Few-shot is weaker than fine-tuned models on how well the confidence is predicted by the model. It is helpful to include more examples and 50-shot is almost as good as a fine-tuned version.',
    'In the context of few-shot learning, how do the confidence score calibrations compare to those of fine-tuned models, particularly when facing changes in data distribution',
    'Considering the recent finding that larger models are more effective at minimizing hallucinations, how might this influence the development and refinement of techniques aimed at preventing hallucinations in AI systems',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9208
cosine_accuracy@3	0.995
cosine_accuracy@5	0.995
cosine_accuracy@10	1.0
cosine_precision@1	0.9208
cosine_precision@3	0.3317
cosine_precision@5	0.199
cosine_precision@10	0.1
cosine_recall@1	0.9208
cosine_recall@3	0.995
cosine_recall@5	0.995
cosine_recall@10	1.0
cosine_ndcg@10	0.9694
cosine_mrr@10	0.9587
cosine_map@100	0.9587

Information Retrieval

• Dataset: dim_512
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9257
cosine_accuracy@3	0.995
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.9257
cosine_precision@3	0.3317
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.9257
cosine_recall@3	0.995
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9716
cosine_mrr@10	0.9616
cosine_map@100	0.9616

Information Retrieval

• Dataset: dim_256
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9158
cosine_accuracy@3	1.0
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.9158
cosine_precision@3	0.3333
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.9158
cosine_recall@3	1.0
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9676
cosine_mrr@10	0.9563
cosine_map@100	0.9563

Information Retrieval

• Dataset: dim_128
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9158
cosine_accuracy@3	0.995
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.9158
cosine_precision@3	0.3317
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.9158
cosine_recall@3	0.995
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9677
cosine_mrr@10	0.9564
cosine_map@100	0.9564

Information Retrieval

• Dataset: dim_64
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.901
cosine_accuracy@3	1.0
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.901
cosine_precision@3	0.3333
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.901
cosine_recall@3	1.0
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9622
cosine_mrr@10	0.9488
cosine_map@100	0.9488

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 5
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• load_best_model_at_end: True

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
0.0220	5	6.6173	-	-	-	-	-
0.0441	10	5.5321	-	-	-	-	-
0.0661	15	5.656	-	-	-	-	-
0.0881	20	4.9256	-	-	-	-	-
0.1101	25	5.0757	-	-	-	-	-
0.1322	30	5.2047	-	-	-	-	-
0.1542	35	5.1307	-	-	-	-	-
0.1762	40	4.9219	-	-	-	-	-
0.1982	45	5.1957	-	-	-	-	-
0.2203	50	5.36	-	-	-	-	-
0.2423	55	3.0865	-	-	-	-	-
0.2643	60	3.7054	-	-	-	-	-
0.2863	65	2.9541	-	-	-	-	-
0.3084	70	3.5521	-	-	-	-	-
0.3304	75	3.5665	-	-	-	-	-
0.3524	80	2.9532	-	-	-	-	-
0.3744	85	2.5121	-	-	-	-	-
0.3965	90	3.1269	-	-	-	-	-
0.4185	95	3.4048	-	-	-	-	-
0.4405	100	2.8126	-	-	-	-	-
0.4626	105	1.6847	-	-	-	-	-
0.4846	110	1.3331	-	-	-	-	-
0.5066	115	2.4799	-	-	-	-	-
0.5286	120	2.1176	-	-	-	-	-
0.5507	125	2.4249	-	-	-	-	-
0.5727	130	3.3705	-	-	-	-	-
0.5947	135	1.551	-	-	-	-	-
0.6167	140	1.328	-	-	-	-	-
0.6388	145	1.9353	-	-	-	-	-
0.6608	150	2.4254	-	-	-	-	-
0.6828	155	1.8436	-	-	-	-	-
0.7048	160	1.1937	-	-	-	-	-
0.7269	165	2.164	-	-	-	-	-
0.7489	170	2.2921	-	-	-	-	-
0.7709	175	2.4385	-	-	-	-	-
0.7930	180	1.2392	-	-	-	-	-
0.8150	185	1.0472	-	-	-	-	-
0.8370	190	1.5844	-	-	-	-	-
0.8590	195	1.2492	-	-	-	-	-
0.8811	200	1.6774	-	-	-	-	-
0.9031	205	2.485	-	-	-	-	-
0.9251	210	2.4781	-	-	-	-	-
0.9471	215	2.4476	-	-	-	-	-
0.9692	220	2.6243	-	-	-	-	-
0.9912	225	1.3651	-	-	-	-	-
1.0	227	-	0.9066	0.9112	0.9257	0.8906	0.9182
1.0132	230	1.0575	-	-	-	-	-
1.0352	235	1.4499	-	-	-	-	-
1.0573	240	1.4333	-	-	-	-	-
1.0793	245	1.1148	-	-	-	-	-
1.1013	250	1.259	-	-	-	-	-
1.1233	255	0.873	-	-	-	-	-
1.1454	260	1.646	-	-	-	-	-
1.1674	265	1.7583	-	-	-	-	-
1.1894	270	1.2268	-	-	-	-	-
1.2115	275	1.3792	-	-	-	-	-
1.2335	280	2.5662	-	-	-	-	-
1.2555	285	1.5021	-	-	-	-	-
1.2775	290	1.1399	-	-	-	-	-
1.2996	295	1.3307	-	-	-	-	-
1.3216	300	0.7458	-	-	-	-	-
1.3436	305	1.1029	-	-	-	-	-
1.3656	310	1.0205	-	-	-	-	-
1.3877	315	1.0998	-	-	-	-	-
1.4097	320	0.8304	-	-	-	-	-
1.4317	325	1.3673	-	-	-	-	-
1.4537	330	2.4445	-	-	-	-	-
1.4758	335	2.8757	-	-	-	-	-
1.4978	340	1.7879	-	-	-	-	-
1.5198	345	1.1255	-	-	-	-	-
1.5419	350	1.6743	-	-	-	-	-
1.5639	355	1.3803	-	-	-	-	-
1.5859	360	1.1998	-	-	-	-	-
1.6079	365	1.2129	-	-	-	-	-
1.6300	370	1.6588	-	-	-	-	-
1.6520	375	0.9827	-	-	-	-	-
1.6740	380	0.605	-	-	-	-	-
1.6960	385	1.2934	-	-	-	-	-
1.7181	390	1.1776	-	-	-	-	-
1.7401	395	1.445	-	-	-	-	-
1.7621	400	0.6393	-	-	-	-	-
1.7841	405	0.9303	-	-	-	-	-
1.8062	410	0.7541	-	-	-	-	-
1.8282	415	0.5413	-	-	-	-	-
1.8502	420	1.5258	-	-	-	-	-
1.8722	425	1.4257	-	-	-	-	-
1.8943	430	1.3111	-	-	-	-	-
1.9163	435	1.6604	-	-	-	-	-
1.9383	440	1.4004	-	-	-	-	-
1.9604	445	2.7186	-	-	-	-	-
1.9824	450	2.2757	-	-	-	-	-
2.0	454	-	0.9401	0.9433	0.9387	0.9386	0.9416
2.0044	455	0.9345	-	-	-	-	-
2.0264	460	0.9325	-	-	-	-	-
2.0485	465	1.2434	-	-	-	-	-
2.0705	470	1.5161	-	-	-	-	-
2.0925	475	2.6011	-	-	-	-	-
2.1145	480	1.8276	-	-	-	-	-
2.1366	485	1.5005	-	-	-	-	-
2.1586	490	0.8618	-	-	-	-	-
2.1806	495	2.1422	-	-	-	-	-
2.2026	500	1.3922	-	-	-	-	-
2.2247	505	1.5939	-	-	-	-	-
2.2467	510	1.3021	-	-	-	-	-
2.2687	515	1.0825	-	-	-	-	-
2.2907	520	0.9066	-	-	-	-	-
2.3128	525	0.7717	-	-	-	-	-
2.3348	530	1.1484	-	-	-	-	-
2.3568	535	1.6513	-	-	-	-	-
2.3789	540	1.7267	-	-	-	-	-
2.4009	545	0.7659	-	-	-	-	-
2.4229	550	2.0213	-	-	-	-	-
2.4449	555	0.5329	-	-	-	-	-
2.4670	560	1.2083	-	-	-	-	-
2.4890	565	1.5432	-	-	-	-	-
2.5110	570	0.5423	-	-	-	-	-
2.5330	575	0.2613	-	-	-	-	-
2.5551	580	0.7985	-	-	-	-	-
2.5771	585	0.3003	-	-	-	-	-
2.5991	590	2.2234	-	-	-	-	-
2.6211	595	0.4772	-	-	-	-	-
2.6432	600	1.0158	-	-	-	-	-
2.6652	605	2.6385	-	-	-	-	-
2.6872	610	0.7042	-	-	-	-	-
2.7093	615	1.1469	-	-	-	-	-
2.7313	620	1.4092	-	-	-	-	-
2.7533	625	0.6487	-	-	-	-	-
2.7753	630	1.218	-	-	-	-	-
2.7974	635	1.1509	-	-	-	-	-
2.8194	640	1.1524	-	-	-	-	-
2.8414	645	0.6477	-	-	-	-	-
2.8634	650	0.6295	-	-	-	-	-
2.8855	655	1.3026	-	-	-	-	-
2.9075	660	1.9196	-	-	-	-	-
2.9295	665	1.3743	-	-	-	-	-
2.9515	670	0.8934	-	-	-	-	-
2.9736	675	1.1801	-	-	-	-	-
2.9956	680	1.2952	-	-	-	-	-
3.0	681	-	0.9538	0.9513	0.9538	0.9414	0.9435
3.0176	685	0.3324	-	-	-	-	-
3.0396	690	0.9551	-	-	-	-	-
3.0617	695	0.9315	-	-	-	-	-
3.0837	700	1.3611	-	-	-	-	-
3.1057	705	1.4406	-	-	-	-	-
3.1278	710	0.5888	-	-	-	-	-
3.1498	715	0.9149	-	-	-	-	-
3.1718	720	0.5627	-	-	-	-	-
3.1938	725	1.6876	-	-	-	-	-
3.2159	730	1.1366	-	-	-	-	-
3.2379	735	1.3571	-	-	-	-	-
3.2599	740	1.5227	-	-	-	-	-
3.2819	745	2.5139	-	-	-	-	-
3.3040	750	0.3735	-	-	-	-	-
3.3260	755	1.4386	-	-	-	-	-
3.3480	760	0.3838	-	-	-	-	-
3.3700	765	0.3973	-	-	-	-	-
3.3921	770	1.4972	-	-	-	-	-
3.4141	775	1.5118	-	-	-	-	-
3.4361	780	0.478	-	-	-	-	-
3.4581	785	1.5982	-	-	-	-	-
3.4802	790	0.6209	-	-	-	-	-
3.5022	795	0.5902	-	-	-	-	-
3.5242	800	1.0877	-	-	-	-	-
3.5463	805	0.9553	-	-	-	-	-
3.5683	810	0.3054	-	-	-	-	-
3.5903	815	1.2229	-	-	-	-	-
3.6123	820	0.7434	-	-	-	-	-
3.6344	825	1.5447	-	-	-	-	-
3.6564	830	1.0751	-	-	-	-	-
3.6784	835	0.8161	-	-	-	-	-
3.7004	840	0.4382	-	-	-	-	-
3.7225	845	1.3547	-	-	-	-	-
3.7445	850	1.7112	-	-	-	-	-
3.7665	855	0.5362	-	-	-	-	-
3.7885	860	0.9309	-	-	-	-	-
3.8106	865	1.8301	-	-	-	-	-
3.8326	870	1.5554	-	-	-	-	-
3.8546	875	1.4035	-	-	-	-	-
3.8767	880	1.5814	-	-	-	-	-
3.8987	885	0.7283	-	-	-	-	-
3.9207	890	1.8549	-	-	-	-	-
3.9427	895	0.196	-	-	-	-	-
3.9648	900	1.2072	-	-	-	-	-
3.9868	905	0.83	-	-	-	-	-
4.0	908	-	0.9564	0.9587	0.9612	0.9488	0.9563
4.0088	910	1.7222	-	-	-	-	-
4.0308	915	0.6728	-	-	-	-	-
4.0529	920	0.9388	-	-	-	-	-
4.0749	925	0.7998	-	-	-	-	-
4.0969	930	1.1561	-	-	-	-	-
4.1189	935	2.4315	-	-	-	-	-
4.1410	940	1.3263	-	-	-	-	-
4.1630	945	1.2374	-	-	-	-	-
4.1850	950	1.1307	-	-	-	-	-
4.2070	955	0.5512	-	-	-	-	-
4.2291	960	1.3266	-	-	-	-	-
4.2511	965	1.2306	-	-	-	-	-
4.2731	970	1.7083	-	-	-	-	-
4.2952	975	0.7028	-	-	-	-	-
4.3172	980	1.2987	-	-	-	-	-
4.3392	985	1.545	-	-	-	-	-
4.3612	990	1.004	-	-	-	-	-
4.3833	995	0.8276	-	-	-	-	-
4.4053	1000	1.4694	-	-	-	-	-
4.4273	1005	0.4914	-	-	-	-	-
4.4493	1010	0.9894	-	-	-	-	-
4.4714	1015	0.8855	-	-	-	-	-
4.4934	1020	1.1339	-	-	-	-	-
4.5154	1025	1.0786	-	-	-	-	-
4.5374	1030	1.2547	-	-	-	-	-
4.5595	1035	0.5312	-	-	-	-	-
4.5815	1040	1.4938	-	-	-	-	-
4.6035	1045	0.8124	-	-	-	-	-
4.6256	1050	1.2401	-	-	-	-	-
4.6476	1055	1.1902	-	-	-	-	-
4.6696	1060	1.4183	-	-	-	-	-
4.6916	1065	1.0718	-	-	-	-	-
4.7137	1070	1.2203	-	-	-	-	-
4.7357	1075	0.8535	-	-	-	-	-
4.7577	1080	1.2454	-	-	-	-	-
4.7797	1085	0.4216	-	-	-	-	-
4.8018	1090	0.8327	-	-	-	-	-
4.8238	1095	1.2371	-	-	-	-	-
4.8458	1100	1.0949	-	-	-	-	-
4.8678	1105	1.2177	-	-	-	-	-
4.8899	1110	0.6236	-	-	-	-	-
4.9119	1115	0.646	-	-	-	-	-
4.9339	1120	1.1822	-	-	-	-	-
4.9559	1125	1.0471	-	-	-	-	-
4.9780	1130	0.7626	-	-	-	-	-
5.0	1135	0.9794	0.9564	0.9563	0.9616	0.9488	0.9587

• The bold row denotes the saved checkpoint.

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.42.4
• PyTorch: 2.3.1+cu121
• Accelerate: 0.32.1
• Datasets: 2.21.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}