metadata

language: []
library_name: sentence-transformers
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - dataset_size:10K<n<100K
  - loss:CoSENTLoss
base_model: sentence-transformers/all-MiniLM-L6-v2
metrics:
  - pearson_cosine
  - spearman_cosine
  - pearson_manhattan
  - spearman_manhattan
  - pearson_euclidean
  - spearman_euclidean
  - pearson_dot
  - spearman_dot
  - pearson_max
  - spearman_max
widget:
  - source_sentence: >-
      Driving or commuting to work feels draining, even if it's a short
      distance.
    sentences:
      - >-
        Symptoms during a manic episode include decreased need for sleep, more
        talkative than usual, flight of ideas, distractibility
      - >-
        I feel like I have lost a part of myself since the traumatic event, and
        I struggle to connect with others on a deeper level.
      - >-
        Diagnosis requires at least one hypomanic episode and one major
        depressive episode.
  - source_sentence: >-
      I felt like my thoughts were disconnected and chaotic during a manic
      episode.
    sentences:
      - >-
        Diagnosis requires one or more manic episodes, which may be preceded or
        followed by hypomanic or major depressive episodes.
      - >-
        I feel like I have lost a part of myself since the traumatic event, and
        I struggle to connect with others on a deeper level.
      - >-
        Depressed mood for most of the day, for more days than not, as indicated
        by subjective account or observation, for at least 2 years.
  - source_sentence: >-
      My insomnia has caused me to experience frequent headaches and muscle
      soreness.
    sentences:
      - Insomnia or hypersomnia nearly every day.
      - >-
        I have difficulty standing in long lines at the grocery store or the
        bank due to the fear of feeling trapped or overwhelmed.
      - >-
        Diagnosis requires at least one hypomanic episode and one major
        depressive episode.
  - source_sentence: >-
      The phobic object or situation almost always provokes immediate fear or
      anxiety.
    sentences:
      - The agoraphobic situations almost always provoke fear or anxiety.
      - >-
        I have difficulty standing in long lines at the grocery store or the
        bank due to the fear of feeling trapped or overwhelmed.
      - >-
        Exclusion of schizoaffective disorder and depressive or bipolar disorder
        with psychotic features, based on the absence of concurrent depressive
        or manic episodes during the active-phase symptoms, or these mood
        episodes being present for a minority of the total duration of the
        active and residual phases.
  - source_sentence: >-
      I engage in risky behaviors like reckless driving or reckless sexual
      encounters.
    sentences:
      - >-
        Symptoms during a manic episode include inflated self-esteem or
        grandiosity,increased goal-directed activity, or excessive involvement
        in risky activities.
      - >-
        Marked decrease in functioning in areas like work, interpersonal
        relations, or self-care since the onset of the disturbance.
      - >-
        During the specified period, symptoms from Criterion A are present at
        least half the time with no symptom-free interval lasting longer than 2
        months.
pipeline_tag: sentence-similarity
model-index:
  - name: SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2
    results:
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: FT label
          type: FT_label
        metrics:
          - type: pearson_cosine
            value: 0.4627701543833943
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.4076356119364853
            name: Spearman Cosine
          - type: pearson_manhattan
            value: 0.48164714740150605
            name: Pearson Manhattan
          - type: spearman_manhattan
            value: 0.406731043246377
            name: Spearman Manhattan
          - type: pearson_euclidean
            value: 0.4840582172096936
            name: Pearson Euclidean
          - type: spearman_euclidean
            value: 0.407636256115058
            name: Spearman Euclidean
          - type: pearson_dot
            value: 0.46277015122653486
            name: Pearson Dot
          - type: spearman_dot
            value: 0.4076359510487126
            name: Spearman Dot
          - type: pearson_max
            value: 0.4840582172096936
            name: Pearson Max
          - type: spearman_max
            value: 0.407636256115058
            name: Spearman Max

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/all-MiniLM-L6-v2
Maximum Sequence Length: 256 tokens
Output Dimensionality: 384 tokens
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Hgkang00/FT-label-consent-20")
# Run inference
sentences = [
    'I engage in risky behaviors like reckless driving or reckless sexual encounters.',
    'Symptoms during a manic episode include inflated self-esteem or grandiosity,increased goal-directed activity, or excessive involvement in risky activities.',
    'Marked decrease in functioning in areas like work, interpersonal relations, or self-care since the onset of the disturbance.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

Dataset: FT_label
Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.4628
spearman_cosine	0.4076
pearson_manhattan	0.4816
spearman_manhattan	0.4067
pearson_euclidean	0.4841
spearman_euclidean	0.4076
pearson_dot	0.4628
spearman_dot	0.4076
pearson_max	0.4841
spearman_max	0.4076

Training Details

Training Dataset

Unnamed Dataset

Size: 33,800 training samples
Columns: sentence1, sentence2, and score
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 score
type string string float
details
min: 11 tokens
mean: 31.63 tokens
max: 63 tokens

min: 14 tokens
mean: 25.22 tokens
max: 41 tokens

min: -1.0
mean: -0.87
max: 1.0

	sentence1	sentence2	score
type	string	string	float
details	min: 11 tokens mean: 31.63 tokens max: 63 tokens	min: 14 tokens mean: 25.22 tokens max: 41 tokens	min: -1.0 mean: -0.87 max: 1.0

Samples:

sentence1	sentence2	score
`Presence of one or more of the following intrusion symptoms associated with the traumatic event: recurrent distressing memories, dreams, flashbacks, psychological distress, or physiological reactions to cues of the traumatic event.`	`I avoid making phone calls, even to close friends or family, because I'm afraid of saying something wrong or sounding awkward.`	`0.0`
`The phobic object or situation almost always provokes immediate fear or anxiety.`	`I find it hard to stick to a consistent eating schedule, sometimes going days without feeling the need to eat at all.`	`-1.0`
`The fear or anxiety is out of proportion to the actual danger posed by the specific object or situation and to the sociocultural context.`	`I have difficulty going to places where I feel there are no immediate exits, such as cinemas or auditoriums, as the fear of being stuck or unable to escape escalates my anxiety.`	`-1.0`

Loss: CoSENTLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "pairwise_cos_sim"
}

Evaluation Dataset

Unnamed Dataset

Size: 4,225 evaluation samples
Columns: sentence1, sentence2, and score
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 score
type string string float
details
min: 11 tokens
mean: 31.24 tokens
max: 63 tokens

min: 15 tokens
mean: 24.86 tokens
max: 41 tokens

min: -1.0
mean: -0.87
max: 1.0

	sentence1	sentence2	score
type	string	string	float
details	min: 11 tokens mean: 31.24 tokens max: 63 tokens	min: 15 tokens mean: 24.86 tokens max: 41 tokens	min: -1.0 mean: -0.87 max: 1.0

Samples:

sentence1	sentence2	score
`Excessive anxiety and worry occurring more days than not for at least 6 months, about a number of events or activities such as work or school performance.`	`Simple activities like going for a walk or doing household chores feel like daunting tasks due to my low energy levels.`	`-1.0`
`The individual fears acting in a way or showing anxiety symptoms that will be negatively evaluated, leading to humiliation, embarrassment, rejection, or offense to others.`	`I often find myself mindlessly snacking throughout the day due to changes in my appetite.`	`-1.0`
`Persistent avoidance of stimuli associated with the trauma, evidenced by avoiding distressing memories, thoughts, or feelings, or external reminders of the event.`	`Simple activities like going for a walk or doing household chores feel like daunting tasks due to my low energy levels.`	`-1.0`

Loss: CoSENTLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "pairwise_cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 128
per_device_eval_batch_size: 128
num_train_epochs: 20
warmup_ratio: 0.1

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 128
per_device_eval_batch_size: 128
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 20
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss	FT_label_spearman_cosine
1.0	265	-	6.9529	0.3450
2.0	530	7.5663	7.1002	0.4103
3.0	795	-	7.4786	0.4155
4.0	1060	5.5492	8.6710	0.4115
5.0	1325	-	10.3786	0.4056
6.0	1590	4.3991	10.4239	0.3987
7.0	1855	-	11.8681	0.4238
8.0	2120	3.5916	13.0752	0.4030
9.0	2385	-	12.8567	0.4240
10.0	2650	3.1139	12.4373	0.4270
11.0	2915	-	13.6725	0.4212
12.0	3180	2.6658	15.0521	0.4134
13.0	3445	-	15.4305	0.4114
14.0	3710	2.2024	15.5511	0.4060
15.0	3975	-	14.9427	0.4165
16.0	4240	1.8955	14.8399	0.4162
17.0	4505	-	15.0070	0.4170
18.0	4770	1.712	15.4417	0.4105
19.0	5035	-	15.6241	0.4086
20.0	5300	1.5088	15.6818	0.4076

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.0.0
Transformers: 4.41.1
PyTorch: 2.3.0+cu121
Accelerate: 0.30.1
Datasets: 2.19.1
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}