manu's picture
Add new SentenceTransformer model.
5f71831 verified
metadata
language:
  - en
library_name: sentence-transformers
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dataset_size:100K<n<1M
  - loss:CachedMultipleNegativesRankingLoss
base_model: Unbabel/xlm-roberta-comet-small
metrics:
  - cosine_accuracy
  - dot_accuracy
  - manhattan_accuracy
  - euclidean_accuracy
  - max_accuracy
widget:
  - source_sentence: There's a dock
    sentences:
      - There is a door.
      - the animal is running
      - The woman is singing.
  - source_sentence: The boy scowls
    sentences:
      - A boy is blowing bubbles.
      - He is playing a song.
      - They are driving cars.
  - source_sentence: A bird flying.
    sentences:
      - A butterfly flys freely.
      - A dog carries a bone.
      - Two dogs are playing.
  - source_sentence: A woman sings.
    sentences:
      - The woman is singing.
      - The man is in a city.
      - there is a man in a pool.
  - source_sentence: a baby smiling
    sentences:
      - A baby is unhappy.
      - The dog has big ears.
      - They are driving cars.
pipeline_tag: sentence-similarity
model-index:
  - name: SentenceTransformer based on Unbabel/xlm-roberta-comet-small
    results:
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: all nli dev
          type: all-nli-dev
        metrics:
          - type: cosine_accuracy
            value: 0.849
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.163
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.837
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.841
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.849
            name: Max Accuracy
      - task:
          type: triplet
          name: Triplet
        dataset:
          name: all nli test
          type: all-nli-test
        metrics:
          - type: cosine_accuracy
            value: 0.839
            name: Cosine Accuracy
          - type: dot_accuracy
            value: 0.15
            name: Dot Accuracy
          - type: manhattan_accuracy
            value: 0.827
            name: Manhattan Accuracy
          - type: euclidean_accuracy
            value: 0.827
            name: Euclidean Accuracy
          - type: max_accuracy
            value: 0.839
            name: Max Accuracy

SentenceTransformer based on Unbabel/xlm-roberta-comet-small

This is a sentence-transformers model finetuned from Unbabel/xlm-roberta-comet-small on the sentence-transformers/all-nli dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("mics-nlp/xlm-roberta-small-all-nli-triplet")
# Run inference
sentences = [
    'a baby smiling',
    'A baby is unhappy.',
    'The dog has big ears.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

Metric Value
cosine_accuracy 0.849
dot_accuracy 0.163
manhattan_accuracy 0.837
euclidean_accuracy 0.841
max_accuracy 0.849

Triplet

Metric Value
cosine_accuracy 0.839
dot_accuracy 0.15
manhattan_accuracy 0.827
euclidean_accuracy 0.827
max_accuracy 0.839

Training Details

Training Dataset

sentence-transformers/all-nli

  • Dataset: sentence-transformers/all-nli at d482672
  • Size: 100,000 training samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 7 tokens
    • mean: 10.9 tokens
    • max: 52 tokens
    • min: 6 tokens
    • mean: 13.62 tokens
    • max: 42 tokens
    • min: 5 tokens
    • mean: 14.76 tokens
    • max: 55 tokens
  • Samples:
    anchor positive negative
    A person on a horse jumps over a broken down airplane. A person is outdoors, on a horse. A person is at a diner, ordering an omelette.
    Children smiling and waving at camera There are children present The kids are frowning
    A boy is jumping on skateboard in the middle of a red bridge. The boy does a skateboarding trick. The boy skates down the sidewalk.
  • Loss: CachedMultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim"
    }
    

Evaluation Dataset

sentence-transformers/all-nli

  • Dataset: sentence-transformers/all-nli at d482672
  • Size: 1,000 evaluation samples
  • Columns: anchor, positive, and negative
  • Approximate statistics based on the first 1000 samples:
    anchor positive negative
    type string string string
    details
    • min: 6 tokens
    • mean: 20.31 tokens
    • max: 83 tokens
    • min: 5 tokens
    • mean: 10.71 tokens
    • max: 35 tokens
    • min: 5 tokens
    • mean: 11.39 tokens
    • max: 32 tokens
  • Samples:
    anchor positive negative
    Two women are embracing while holding to go packages. Two woman are holding packages. The men are fighting outside a deli.
    Two young children in blue jerseys, one with the number 9 and one with the number 2 are standing on wooden steps in a bathroom and washing their hands in a sink. Two kids in numbered jerseys wash their hands. Two kids in jackets walk to school.
    A man selling donuts to a customer during a world exhibition event held in the city of Angeles A man selling donuts to a customer. A woman drinks her coffee in a small cafe.
  • Loss: CachedMultipleNegativesRankingLoss with these parameters:
    {
        "scale": 20.0,
        "similarity_fct": "cos_sim"
    }
    

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • num_train_epochs: 1
  • warmup_ratio: 0.1
  • bf16: True
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 1
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss all-nli-dev_max_accuracy all-nli-test_max_accuracy
0 0 - - 0.541 -
0.016 100 3.5308 3.1817 0.558 -
0.032 200 3.2784 3.0406 0.597 -
0.048 300 3.113 2.7572 0.635 -
0.064 400 2.8296 2.4646 0.68 -
0.08 500 2.631 2.3583 0.676 -
0.096 600 2.3247 2.1394 0.706 -
0.112 700 2.2211 2.0201 0.711 -
0.128 800 2.1263 1.9560 0.757 -
0.144 900 2.2105 1.9074 0.748 -
0.16 1000 2.0637 1.9289 0.728 -
0.176 1100 2.1772 1.8796 0.741 -
0.192 1200 2.1518 1.8346 0.761 -
0.208 1300 1.728 1.8213 0.765 -
0.224 1400 1.8101 1.6321 0.772 -
0.24 1500 1.7516 1.5669 0.793 -
0.256 1600 1.4988 1.5538 0.8 -
0.272 1700 1.6695 1.5462 0.803 -
0.288 1800 1.5971 1.5499 0.783 -
0.304 1900 1.5614 1.5047 0.788 -
0.32 2000 1.522 1.4957 0.794 -
0.336 2100 1.3624 1.4153 0.814 -
0.352 2200 1.4773 1.4169 0.809 -
0.368 2300 1.6066 1.3697 0.813 -
0.384 2400 1.5106 1.3203 0.819 -
0.4 2500 1.4783 1.3417 0.817 -
0.416 2600 1.3696 1.2650 0.824 -
0.432 2700 1.5115 1.2779 0.829 -
0.448 2800 1.4834 1.2668 0.834 -
0.464 2900 1.4823 1.2621 0.836 -
0.48 3000 1.4163 1.2465 0.837 -
0.496 3100 1.4232 1.2475 0.837 -
0.512 3200 1.2193 1.1975 0.838 -
0.528 3300 1.2569 1.1816 0.838 -
0.544 3400 1.2988 1.1936 0.839 -
0.56 3500 1.5068 1.2213 0.835 -
0.576 3600 1.3022 1.1799 0.842 -
0.592 3700 1.3823 1.1910 0.831 -
0.608 3800 1.4224 1.1786 0.834 -
0.624 3900 1.3765 1.1541 0.843 -
0.64 4000 1.4987 1.1365 0.844 -
0.656 4100 1.7525 1.1394 0.843 -
0.672 4200 1.6013 1.1178 0.841 -
0.688 4300 1.3326 1.0959 0.846 -
0.704 4400 1.355 1.0757 0.848 -
0.72 4500 1.2834 1.0681 0.846 -
0.736 4600 1.2939 1.0696 0.85 -
0.752 4700 1.4069 1.0645 0.848 -
0.768 4800 1.4503 1.0609 0.849 -
0.784 4900 1.2833 1.0587 0.847 -
0.8 5000 1.3321 1.0563 0.849 -
0.816 5100 1.3006 1.0539 0.847 -
0.832 5200 1.4332 1.0527 0.847 -
0.848 5300 1.3101 1.0505 0.848 -
0.864 5400 1.3658 1.0523 0.849 -
0.88 5500 1.353 1.0520 0.849 -
0.896 5600 1.2429 1.0521 0.848 -
0.912 5700 1.3512 1.0505 0.848 -
0.928 5800 1.2995 1.0501 0.848 -
0.944 5900 1.3514 1.0491 0.849 -
0.96 6000 1.3976 1.0490 0.848 -
0.976 6100 1.2112 1.0487 0.848 -
0.992 6200 0.0033 1.0492 0.849 -
1.0 6250 - - - 0.839

Framework Versions

  • Python: 3.9.10
  • Sentence Transformers: 3.0.0
  • Transformers: 4.41.2
  • PyTorch: 2.3.0+cu121
  • Accelerate: 0.26.1
  • Datasets: 2.16.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, 
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}