tomaarsen's picture
tomaarsen HF staff
Update README.md
e88732e verified
metadata
language:
  - en
license: apache-2.0
library_name: sentence-transformers
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - 100K<n<1M
  - loss:MultipleNegativesRankingLoss
base_model: microsoft/mpnet-base
metrics:
  - cosine_accuracy
  - dot_accuracy
  - manhattan_accuracy
  - euclidean_accuracy
  - max_accuracy
widget:
  - source_sentence: >-
      The strangely dressed guys, one wearing an orange wig, sunglasses with
      peace signs, and a karate costume with an orannge belt, another wearing a
      curly blue wig, heart shaped sunglasses, and a karate outfit painted with
      leaves, and the third wearing pink underwear, a black afro, and giant
      sunglasses.
    sentences:
      - >-
        A blonde female is reaching into a golf hole while holding two golf
        balls.
      - There are people wearing outfits.
      - The people are naked.
  - source_sentence: A group of children playing and having a good time.
    sentences:
      - The kids are together.
      - The children are reading books.
      - People are pointing at a Middle-aged woman.
  - source_sentence: >-
      Three children dressed in winter clothes are walking through the woods
      while pushing cargo along.
    sentences:
      - A woman is sitting.
      - Three childre are dressed in summer clothes.
      - Three children are dressed in winter clothes.
  - source_sentence: A young child is enjoying the water and rock scenery with their dog.
    sentences:
      - The child and dog are enjoying some fresh air.
      - The teenage boy is taking his cat for a walk beside the water.
      - A lady in blue has birds around her.
  - source_sentence: >-
      Boca da Corrida Encumeada (moderate; 5 hours): views of Curral das Freiras
      and the valley of Ribeiro do Poco.
    sentences:
      - >-
        Boca da Corrida Encumeada is a moderate text that takes 5 hours to
        complete. 
      - This chapter is in the advance category.
      - I think it is something that we need.
pipeline_tag: sentence-similarity
co2_eq_emissions:
  emissions: 118.81134392463773
  energy_consumed: 0.30566177669432554
  source: codecarbon
  training_type: fine-tuning
  on_cloud: false
  cpu_model: 13th Gen Intel(R) Core(TM) i7-13700K
  ram_total_size: 31.777088165283203
  hours_used: 1.661
  hardware_used: 1 x NVIDIA GeForce RTX 3090
model-index:
  - name: MPNet base trained on AllNLI triplets
    results:
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: all nli dev
          type: all-nli-dev
        metrics:
          - type: cosine_accuracy
            value: 0.9003645200486027
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.09705346294046173
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.8968712029161604
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.8974787363304981
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.9003645200486027
            name: Max Accuracy
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: all nli test
          type: all-nli-test
        metrics:
          - type: cosine_accuracy
            value: 0.9149644424269935
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.08564079285822364
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.911484339536995
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.9134513542139506
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.9149644424269935
            name: Max Accuracy

MPNet base trained on AllNLI triplets

This is a sentence-transformers model finetuned from microsoft/mpnet-base on the sentence-transformers/all-nli dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: microsoft/mpnet-base
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity
  • Training Dataset:
  • Language: en
  • License: apache-2.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/mpnet-base-all-nli-triplet")
# Run inference
sentences = [
    'Then he ran.',
    'The people are running.',
    'The man is on his bike.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

Metric Value
cosine_accuracy 0.9004
dot_accuracy 0.0971
manhattan_accuracy 0.8969
euclidean_accuracy 0.8975
max_accuracy 0.9004

Triplet

Metric Value
cosine_accuracy 0.915
dot_accuracy 0.0856
manhattan_accuracy 0.9115
euclidean_accuracy 0.9135
max_accuracy 0.915

Training Details

Training Dataset

sentence-transformers/all-nli

  • Dataset: sentence-transformers/all-nli at d482672
  • Size: 100,000 training samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 7 tokens
    • mean: 10.46 tokens
    • max: 46 tokens
    • min: 6 tokens
    • mean: 12.81 tokens
    • max: 40 tokens
    • min: 5 tokens
    • mean: 13.4 tokens
    • max: 50 tokens
  • Samples:
    anchor positive negative
    A person on a horse jumps over a broken down airplane. A person is outdoors, on a horse. A person is at a diner, ordering an omelette.
    Children smiling and waving at camera There are children present The kids are frowning
    A boy is jumping on skateboard in the middle of a red bridge. The boy does a skateboarding trick. The boy skates down the sidewalk.
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim"
    }
    

Evaluation Dataset

sentence-transformers/all-nli

  • Dataset: sentence-transformers/all-nli at d482672
  • Size: 6,584 evaluation samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 6 tokens
    • mean: 17.95 tokens
    • max: 63 tokens
    • min: 4 tokens
    • mean: 9.78 tokens
    • max: 29 tokens
    • min: 5 tokens
    • mean: 10.35 tokens
    • max: 29 tokens
  • Samples:
    anchor positive negative
    Two women are embracing while holding to go packages. Two woman are holding packages. The men are fighting outside a deli.
    Two young children in blue jerseys, one with the number 9 and one with the number 2 are standing on wooden steps in a bathroom and washing their hands in a sink. Two kids in numbered jerseys wash their hands. Two kids in jackets walk to school.
    A man selling donuts to a customer during a world exhibition event held in the city of Angeles A man selling donuts to a customer. A woman drinks her coffee in a small cafe.
  • Loss: MultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim"
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • num_train_epochs: 1
  • warmup_ratio: 0.1
  • fp16: True
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss all-nli-dev_max_accuracy all-nli-test_max_accuracy
0 0 - - 0.6832 -
0.016 100 2.6355 1.0725 0.7924 -
0.032 200 0.9206 0.8342 0.8080 -
0.048 300 1.2567 0.7855 0.8133 -
0.064 400 0.7949 0.8857 0.7974 -
0.08 500 0.7583 0.9487 0.7872 -
0.096 600 1.0022 1.1312 0.7848 -
0.112 700 0.8178 1.2282 0.7895 -
0.128 800 0.9997 1.5132 0.7488 -
0.144 900 1.1173 1.4605 0.7473 -
0.16 1000 1.0089 1.3794 0.7543 -
0.176 1100 1.0235 1.4188 0.7640 -
0.192 1200 1.0031 1.2465 0.7570 -
0.208 1300 0.8286 1.4176 0.7426 -
0.224 1400 0.8411 1.1914 0.7600 -
0.24 1500 0.8389 1.1719 0.7820 -
0.256 1600 0.7144 1.1167 0.7691 -
0.272 1700 0.881 1.0747 0.7902 -
0.288 1800 0.8657 1.1576 0.7966 -
0.304 1900 0.7323 1.0122 0.8322 -
0.32 2000 0.6578 1.1248 0.8273 -
0.336 2100 0.6037 1.1194 0.8269 -
0.352 2200 0.641 1.1410 0.8341 -
0.368 2300 0.7843 1.0600 0.8328 -
0.384 2400 0.8222 0.9988 0.8161 -
0.4 2500 0.7287 1.2026 0.8395 -
0.416 2600 0.6035 0.8802 0.8273 -
0.432 2700 0.8275 1.1631 0.8458 -
0.448 2800 0.8483 0.9218 0.8316 -
0.464 2900 0.8813 1.1187 0.8147 -
0.48 3000 0.7408 0.9582 0.8246 -
0.496 3100 0.7886 0.9364 0.8261 -
0.512 3200 0.6064 0.8338 0.8302 -
0.528 3300 0.6415 0.7895 0.8650 -
0.544 3400 0.5766 0.7525 0.8571 -
0.56 3500 0.6212 0.8605 0.8572 -
0.576 3600 0.5773 0.7460 0.8419 -
0.592 3700 0.6104 0.7480 0.8580 -
0.608 3800 0.5754 0.7215 0.8657 -
0.624 3900 0.5525 0.7900 0.8630 -
0.64 4000 0.7802 0.7443 0.8612 -
0.656 4100 0.9796 0.7756 0.8748 -
0.672 4200 0.9355 0.6917 0.8796 -
0.688 4300 0.7081 0.6442 0.8832 -
0.704 4400 0.6868 0.6395 0.8891 -
0.72 4500 0.5964 0.5983 0.8820 -
0.736 4600 0.6618 0.5754 0.8861 -
0.752 4700 0.6957 0.6177 0.8803 -
0.768 4800 0.6375 0.5577 0.8881 -
0.784 4900 0.5481 0.5496 0.8835 -
0.8 5000 0.6626 0.5728 0.8949 -
0.816 5100 0.5192 0.5329 0.8935 -
0.832 5200 0.5856 0.5188 0.8935 -
0.848 5300 0.5142 0.5252 0.8920 -
0.864 5400 0.6404 0.5641 0.8885 -
0.88 5500 0.5466 0.5209 0.8929 -
0.896 5600 0.575 0.5170 0.8961 -
0.912 5700 0.626 0.5095 0.9001 -
0.928 5800 0.5631 0.4817 0.8984 -
0.944 5900 0.7301 0.4996 0.8984 -
0.96 6000 0.7712 0.5160 0.9014 -
0.976 6100 0.6203 0.5000 0.9007 -
0.992 6200 0.0005 0.4996 0.9004 -
1.0 6250 - - - 0.9150

Environmental Impact

Carbon emissions were measured using CodeCarbon.

  • Energy Consumed: 0.306 kWh
  • Carbon Emitted: 0.119 kg of CO2
  • Hours Used: 1.661 hours

Training Hardware

  • On Cloud: No
  • GPU Model: 1 x NVIDIA GeForce RTX 3090
  • CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
  • RAM Size: 31.78 GB

Framework Versions

  • Python: 3.11.6
  • Sentence Transformers: 3.0.0.dev0
  • Transformers: 4.41.1
  • PyTorch: 2.3.0+cu121
  • Accelerate: 0.30.1
  • Datasets: 2.19.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}