--- language: - en license: apache-2.0 tags: - sentence-transformers - sentence-similarity - feature-extraction - generated_from_trainer - dataset_size:6300 - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss base_model: BAAI/bge-base-en-v1.5 widget: - source_sentence: We enter into forward currency contracts in order to hedge a portion of the foreign currency exposure associated with the translation of our net investment in our Canadian subsidiary. sentences: - How much did Delta Air Lines spend on debt and finance lease obligations in 2023? - What mechanisms does the company use to hedge foreign currency exposure for its Canadian subsidiary? - How did operating overhead expenses change for NIKE from fiscal 2022 to fiscal 2023? - source_sentence: We calculate return on invested hat capital (ROIC) by dividing adjusted ROIC operating profit for the prior four quarters by the average invested capital. sentences: - What was the fair value of U.S. government and agency securities as of June 30, 2022? - How is the Return on Invested Capital (ROIC) calculated? - What business outcomes is HPE focused on accelerating with its technological solutions? - source_sentence: Expenses from our comparable owned and leased hotels increased $137 million, on a currency neutral basis, as a result of increased occupancy and cost inflation both driving higher labor costs, utilities and other operating expenses, as well as an increase in rent expense. sentences: - How did the expenses from comparable owned and leased hotels change and what were the contributing factors? - What do environmental laws require from suppliers in terms of operations? - What energy management technologies does the Enphase bidirectional EV charger integrate with? - source_sentence: The Advancing Agility & Automation Initiative at The Hershey Company is projected to result in total pre-tax costs of $200,000 to $250,000 from inception through 2026. This includes costs for program office execution and third-party costs supporting the design and implementation of the new organizational structure, as well as implementation and technology capability costs and employee severance and related separation benefits. sentences: - What was the total amortization expense for The Hershey Company in 2021? - How much did net cash used in financing activities decrease in fiscal 2023 compared to the previous fiscal year? - What is the total projected pre-tax cost of The Hershey Company's Advancing Agility & Automation Initiative through 2026? - source_sentence: Structural costs typically do not have a directly proportionate relationship to production volume and include costs such as manufacturing, engineering, and administrative expenses. These costs can be adjusted over time in response to external factors. sentences: - How does Ford Motor Company handle its structural costs in relation to production volume changes? - What were the total future minimum lease payments under all non-cancelable operating leases for the company as of December 31, 2023? - What guidelines does the FASB provide for the measurement of fair value when quoted prices are not available? pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_precision@3 - cosine_precision@5 - cosine_precision@10 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 - cosine_map@100 model-index: - name: BGE base Financial Matryoshka results: - task: type: information-retrieval name: Information Retrieval dataset: name: dim 768 type: dim_768 metrics: - type: cosine_accuracy@1 value: 0.72 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8257142857142857 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8585714285714285 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8942857142857142 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.72 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2752380952380953 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.1717142857142857 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.08942857142857143 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.72 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8257142857142857 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8585714285714285 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8942857142857142 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.8077694527772951 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7800079365079364 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7837848752496734 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 512 type: dim_512 metrics: - type: cosine_accuracy@1 value: 0.7157142857142857 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8242857142857143 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8642857142857143 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8914285714285715 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7157142857142857 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2747619047619047 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17285714285714285 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.08914285714285713 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.7157142857142857 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8242857142857143 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8642857142857143 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8914285714285715 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.805259563189015 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7773735827664396 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7813006780341183 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 256 type: dim_256 metrics: - type: cosine_accuracy@1 value: 0.7028571428571428 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8171428571428572 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8542857142857143 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8814285714285715 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.7028571428571428 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2723809523809524 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17085714285714285 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.08814285714285712 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.7028571428571428 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8171428571428572 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8542857142857143 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8814285714285715 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7945503213768784 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7664075963718817 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7709929668571353 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 128 type: dim_128 metrics: - type: cosine_accuracy@1 value: 0.6785714285714286 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.8028571428571428 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8542857142857143 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8814285714285715 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6785714285714286 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.26761904761904765 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.17085714285714285 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.08814285714285712 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6785714285714286 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.8028571428571428 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8542857142857143 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8814285714285715 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.7829387132685872 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7509529478458048 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7549309056916426 name: Cosine Map@100 - task: type: information-retrieval name: Information Retrieval dataset: name: dim 64 type: dim_64 metrics: - type: cosine_accuracy@1 value: 0.6485714285714286 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.77 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.8142857142857143 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.8657142857142858 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.6485714285714286 name: Cosine Precision@1 - type: cosine_precision@3 value: 0.2566666666666667 name: Cosine Precision@3 - type: cosine_precision@5 value: 0.16285714285714287 name: Cosine Precision@5 - type: cosine_precision@10 value: 0.08657142857142856 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.6485714285714286 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.77 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.8142857142857143 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.8657142857142858 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.755512484642688 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.7203905895691608 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.7247515061294347 name: Cosine Map@100 --- # BGE base Financial Matryoshka This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) - **Maximum Sequence Length:** 512 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - json - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("aired/bge-base-financial-matryoshka") # Run inference sentences = [ 'Structural costs typically do not have a directly proportionate relationship to production volume and include costs such as manufacturing, engineering, and administrative expenses. These costs can be adjusted over time in response to external factors.', 'How does Ford Motor Company handle its structural costs in relation to production volume changes?', 'What were the total future minimum lease payments under all non-cancelable operating leases for the company as of December 31, 2023?', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] ``` ## Evaluation ### Metrics #### Information Retrieval * Datasets: `dim_768`, `dim_512`, `dim_256`, `dim_128` and `dim_64` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | dim_768 | dim_512 | dim_256 | dim_128 | dim_64 | |:--------------------|:-----------|:-----------|:-----------|:-----------|:-----------| | cosine_accuracy@1 | 0.72 | 0.7157 | 0.7029 | 0.6786 | 0.6486 | | cosine_accuracy@3 | 0.8257 | 0.8243 | 0.8171 | 0.8029 | 0.77 | | cosine_accuracy@5 | 0.8586 | 0.8643 | 0.8543 | 0.8543 | 0.8143 | | cosine_accuracy@10 | 0.8943 | 0.8914 | 0.8814 | 0.8814 | 0.8657 | | cosine_precision@1 | 0.72 | 0.7157 | 0.7029 | 0.6786 | 0.6486 | | cosine_precision@3 | 0.2752 | 0.2748 | 0.2724 | 0.2676 | 0.2567 | | cosine_precision@5 | 0.1717 | 0.1729 | 0.1709 | 0.1709 | 0.1629 | | cosine_precision@10 | 0.0894 | 0.0891 | 0.0881 | 0.0881 | 0.0866 | | cosine_recall@1 | 0.72 | 0.7157 | 0.7029 | 0.6786 | 0.6486 | | cosine_recall@3 | 0.8257 | 0.8243 | 0.8171 | 0.8029 | 0.77 | | cosine_recall@5 | 0.8586 | 0.8643 | 0.8543 | 0.8543 | 0.8143 | | cosine_recall@10 | 0.8943 | 0.8914 | 0.8814 | 0.8814 | 0.8657 | | **cosine_ndcg@10** | **0.8078** | **0.8053** | **0.7946** | **0.7829** | **0.7555** | | cosine_mrr@10 | 0.78 | 0.7774 | 0.7664 | 0.751 | 0.7204 | | cosine_map@100 | 0.7838 | 0.7813 | 0.771 | 0.7549 | 0.7248 | ## Training Details ### Training Dataset #### json * Dataset: json * Size: 6,300 training samples * Columns: positive and anchor * Approximate statistics based on the first 1000 samples: | | positive | anchor | |:--------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------| | type | string | string | | details | | | * Samples: | positive | anchor | |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------| | GEICO markets its policies mainly by direct response methods where most customers apply for coverage directly to the company via the Internet or over the telephone. | What are the primary marketing methods used by GEICO? | | In addition, most group health plans and issuers of group or individual health insurance coverage are required to disclose personalized pricing information to their participants, beneficiaries, and enrollees through an online consumer tool, by phone, or in paper form, upon request. Cost estimates must be provided in real-time based on cost-sharing information that is accurate at the time of the request. | What are the requirements for health insurers and group health plans in providing cost estimates to consumers? | | Fair values of indefinite-lived intangible assets are determined based on the income approach. | What method is used to determine the fair value of indefinite-lived intangible assets? | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "MultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: epoch - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 16 - `learning_rate`: 2e-05 - `num_train_epochs`: 4 - `lr_scheduler_type`: cosine - `warmup_ratio`: 0.1 - `fp16`: True - `load_best_model_at_end`: True - `optim`: adamw_torch_fused - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: epoch - `prediction_loss_only`: True - `per_device_train_batch_size`: 32 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 16 - `eval_accumulation_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 4 - `max_steps`: -1 - `lr_scheduler_type`: cosine - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: False - `fp16`: True - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch_fused - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: False - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | dim_768_cosine_ndcg@10 | dim_512_cosine_ndcg@10 | dim_256_cosine_ndcg@10 | dim_128_cosine_ndcg@10 | dim_64_cosine_ndcg@10 | |:----------:|:------:|:-------------:|:----------------------:|:----------------------:|:----------------------:|:----------------------:|:---------------------:| | 0.8122 | 10 | 1.6045 | - | - | - | - | - | | 0.9746 | 12 | - | 0.7895 | 0.7895 | 0.7764 | 0.7680 | 0.7277 | | 1.6244 | 20 | 0.6975 | - | - | - | - | - | | 1.9492 | 24 | - | 0.8044 | 0.8026 | 0.7924 | 0.7819 | 0.7515 | | 2.4365 | 30 | 0.4732 | - | - | - | - | - | | 2.9239 | 36 | - | 0.8064 | 0.8060 | 0.7944 | 0.7825 | 0.7549 | | 3.2487 | 40 | 0.4182 | - | - | - | - | - | | **3.8985** | **48** | **-** | **0.8078** | **0.8053** | **0.7946** | **0.7829** | **0.7555** | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.12 - Sentence Transformers: 3.3.1 - Transformers: 4.41.2 - PyTorch: 2.1.2+cu121 - Accelerate: 1.1.1 - Datasets: 2.19.1 - Tokenizers: 0.19.1 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### MultipleNegativesRankingLoss ```bibtex @misc{henderson2017efficient, title={Efficient Natural Language Response Suggestion for Smart Reply}, author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, year={2017}, eprint={1705.00652}, archivePrefix={arXiv}, primaryClass={cs.CL} } ```