checkpoint-816/README.md

metadata

base_model: colorfulscoop/sbert-base-ja
library_name: sentence-transformers
metrics:
  - cosine_accuracy
  - cosine_accuracy_threshold
  - cosine_f1
  - cosine_f1_threshold
  - cosine_precision
  - cosine_recall
  - cosine_ap
  - dot_accuracy
  - dot_accuracy_threshold
  - dot_f1
  - dot_f1_threshold
  - dot_precision
  - dot_recall
  - dot_ap
  - manhattan_accuracy
  - manhattan_accuracy_threshold
  - manhattan_f1
  - manhattan_f1_threshold
  - manhattan_precision
  - manhattan_recall
  - manhattan_ap
  - euclidean_accuracy
  - euclidean_accuracy_threshold
  - euclidean_f1
  - euclidean_f1_threshold
  - euclidean_precision
  - euclidean_recall
  - euclidean_ap
  - max_accuracy
  - max_accuracy_threshold
  - max_f1
  - max_f1_threshold
  - max_precision
  - max_recall
  - max_ap
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:680
  - loss:CoSENTLoss
widget:
  - source_sentence: 中を見てみよう
    sentences:
      - 外を調べよう
      - リリアンはどんな魔法が使えるの？
      - 花がぬいぐるみに変えられている
  - source_sentence: キャンドル要らない
    sentences:
      - なんで猫が話せる？
      - 自分でやれば？
      - 中を見てみよう
  - source_sentence: 信用できない
    sentences:
      - どっちでもいいよ
      - 誰？
      - 誰かが呪文で花をぬいぐるみに変えた
  - source_sentence: 例えば？
    sentences:
      - 誰かがが魔法をかけた
      - ジャック
      - なんでしなきゃいけないの？
  - source_sentence: 魔法を使える人
    sentences:
      - かっこいいね
      - 物の姿を変えられる人
      - 町って？
model-index:
  - name: SentenceTransformer based on colorfulscoop/sbert-base-ja
    results:
      - task:
          type: binary-classification
          name: Binary Classification
        dataset:
          name: custom arc semantics data jp
          type: custom-arc-semantics-data-jp
        metrics:
          - type: cosine_accuracy
            value: 0.9044117647058824
            name: Cosine Accuracy
          - type: cosine_accuracy_threshold
            value: 0.5501536726951599
            name: Cosine Accuracy Threshold
          - type: cosine_f1
            value: 0.912751677852349
            name: Cosine F1
          - type: cosine_f1_threshold
            value: 0.4790937304496765
            name: Cosine F1 Threshold
          - type: cosine_precision
            value: 0.918918918918919
            name: Cosine Precision
          - type: cosine_recall
            value: 0.9066666666666666
            name: Cosine Recall
          - type: cosine_ap
            value: 0.9084179566135925
            name: Cosine Ap
          - type: dot_accuracy
            value: 0.9117647058823529
            name: Dot Accuracy
          - type: dot_accuracy_threshold
            value: 294.13421630859375
            name: Dot Accuracy Threshold
          - type: dot_f1
            value: 0.9166666666666666
            name: Dot F1
          - type: dot_f1_threshold
            value: 294.13421630859375
            name: Dot F1 Threshold
          - type: dot_precision
            value: 0.9565217391304348
            name: Dot Precision
          - type: dot_recall
            value: 0.88
            name: Dot Recall
          - type: dot_ap
            value: 0.915716305189008
            name: Dot Ap
          - type: manhattan_accuracy
            value: 0.9044117647058824
            name: Manhattan Accuracy
          - type: manhattan_accuracy_threshold
            value: 482.6566162109375
            name: Manhattan Accuracy Threshold
          - type: manhattan_f1
            value: 0.913907284768212
            name: Manhattan F1
          - type: manhattan_f1_threshold
            value: 532.9744262695312
            name: Manhattan F1 Threshold
          - type: manhattan_precision
            value: 0.9078947368421053
            name: Manhattan Precision
          - type: manhattan_recall
            value: 0.92
            name: Manhattan Recall
          - type: manhattan_ap
            value: 0.9104676924615509
            name: Manhattan Ap
          - type: euclidean_accuracy
            value: 0.9117647058823529
            name: Euclidean Accuracy
          - type: euclidean_accuracy_threshold
            value: 23.818954467773438
            name: Euclidean Accuracy Threshold
          - type: euclidean_f1
            value: 0.918918918918919
            name: Euclidean F1
          - type: euclidean_f1_threshold
            value: 23.818954467773438
            name: Euclidean F1 Threshold
          - type: euclidean_precision
            value: 0.9315068493150684
            name: Euclidean Precision
          - type: euclidean_recall
            value: 0.9066666666666666
            name: Euclidean Recall
          - type: euclidean_ap
            value: 0.9093211275077335
            name: Euclidean Ap
          - type: max_accuracy
            value: 0.9117647058823529
            name: Max Accuracy
          - type: max_accuracy_threshold
            value: 482.6566162109375
            name: Max Accuracy Threshold
          - type: max_f1
            value: 0.918918918918919
            name: Max F1
          - type: max_f1_threshold
            value: 532.9744262695312
            name: Max F1 Threshold
          - type: max_precision
            value: 0.9565217391304348
            name: Max Precision
          - type: max_recall
            value: 0.92
            name: Max Recall
          - type: max_ap
            value: 0.915716305189008
            name: Max Ap

SentenceTransformer based on colorfulscoop/sbert-base-ja

This is a sentence-transformers model finetuned from colorfulscoop/sbert-base-ja on the csv dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: colorfulscoop/sbert-base-ja
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • csv

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '魔法を使える人',
    '物の姿を変えられる人',
    'かっこいいね',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Binary Classification

• Dataset: custom-arc-semantics-data-jp
• Evaluated with BinaryClassificationEvaluator

Metric	Value
cosine_accuracy	0.9044
cosine_accuracy_threshold	0.5502
cosine_f1	0.9128
cosine_f1_threshold	0.4791
cosine_precision	0.9189
cosine_recall	0.9067
cosine_ap	0.9084
dot_accuracy	0.9118
dot_accuracy_threshold	294.1342
dot_f1	0.9167
dot_f1_threshold	294.1342
dot_precision	0.9565
dot_recall	0.88
dot_ap	0.9157
manhattan_accuracy	0.9044
manhattan_accuracy_threshold	482.6566
manhattan_f1	0.9139
manhattan_f1_threshold	532.9744
manhattan_precision	0.9079
manhattan_recall	0.92
manhattan_ap	0.9105
euclidean_accuracy	0.9118
euclidean_accuracy_threshold	23.819
euclidean_f1	0.9189
euclidean_f1_threshold	23.819
euclidean_precision	0.9315
euclidean_recall	0.9067
euclidean_ap	0.9093
max_accuracy	0.9118
max_accuracy_threshold	482.6566
max_f1	0.9189
max_f1_threshold	532.9744
max_precision	0.9565
max_recall	0.92
max_ap	0.9157

Training Details

Training Dataset

csv

• Dataset: csv
• Size: 680 training samples
• Columns: text1, text2, and label
• Approximate statistics based on the first 680 samples:

  	text1	text2	label
  type	string	string	int
  details	min: 4 tokens mean: 8.29 tokens max: 15 tokens	min: 4 tokens mean: 7.97 tokens max: 14 tokens	0: ~40.44% 1: ~59.56%

• Samples:

  text1	text2	label
  いらない	うんよろしく	0
  足元よりも更に深くってどこ？	足元よりも更に深くってなに？	1
  他にはないの？	どう思う？	0

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Evaluation Dataset

csv

• Dataset: csv
• Size: 680 evaluation samples
• Columns: text1, text2, and label
• Approximate statistics based on the first 680 samples:

  	text1	text2	label
  type	string	string	int
  details	min: 4 tokens mean: 8.32 tokens max: 15 tokens	min: 4 tokens mean: 8.16 tokens max: 14 tokens	0: ~44.85% 1: ~55.15%

• Samples:

  text1	text2	label
  井戸から水をくんでいた	井戸を使っていた	1
  夕飯は何だったの？	チキンヌードル食べた？	0
  水を井戸からくんでいた	夜ごはんの前	0

• Loss: CoSENTLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "pairwise_cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• learning_rate: 2e-05
• num_train_epochs: 13
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 13
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss	custom-arc-semantics-data-jp_max_ap
None	0	-	-	0.8596
1.0	68	2.6802	1.7807	0.8872
2.0	136	1.4014	1.7683	0.8945
3.0	204	0.7937	1.9877	0.9039
4.0	272	0.5443	1.9106	0.9075
5.0	340	0.4225	1.9418	0.9109
6.0	408	0.3347	2.0123	0.9107
7.0	476	0.3425	2.0387	0.9094
8.0	544	0.2427	1.9878	0.9103
9.0	612	0.2412	2.0424	0.9178
10.0	680	0.1623	2.0273	0.9188
11.0	748	0.1909	2.0955	0.9220
12.0	816	0.1507	2.2124	0.9157

Framework Versions

• Python: 3.10.14
• Sentence Transformers: 3.1.0
• Transformers: 4.44.2
• PyTorch: 2.4.1+cu121
• Accelerate: 0.34.2
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}