Hugging Face's logo

LeoChiuu
/
sbert-base-ja-arc

TensorBoard
Safetensors
Japanese
bert
Model card Files Metrics Community
sbert-base-ja-arc / checkpoint-884 /README.md
LeoChiuu's picture
LeoChiuu
Add new SentenceTransformer model.
e0c1c90 verified
preview code
|
raw
history blame
No virus
20.1 kB
metadata
base_model: colorfulscoop/sbert-base-ja
library_name: sentence-transformers
metrics:
  - cosine_accuracy
  - cosine_accuracy_threshold
  - cosine_f1
  - cosine_f1_threshold
  - cosine_precision
  - cosine_recall
  - cosine_ap
  - dot_accuracy
  - dot_accuracy_threshold
  - dot_f1
  - dot_f1_threshold
  - dot_precision
  - dot_recall
  - dot_ap
  - manhattan_accuracy
  - manhattan_accuracy_threshold
  - manhattan_f1
  - manhattan_f1_threshold
  - manhattan_precision
  - manhattan_recall
  - manhattan_ap
  - euclidean_accuracy
  - euclidean_accuracy_threshold
  - euclidean_f1
  - euclidean_f1_threshold
  - euclidean_precision
  - euclidean_recall
  - euclidean_ap
  - max_accuracy
  - max_accuracy_threshold
  - max_f1
  - max_f1_threshold
  - max_precision
  - max_recall
  - max_ap
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:680
  - loss:CoSENTLoss
widget:
  - source_sentence: 中を見てみよう
    sentences:
      - 外を調べよう
      - リリアンはどんな魔法が使えるの？
      - 花がぬいぐるみに変えられている
  - source_sentence: キャンドル要らない
    sentences:
      - なんで猫が話せる？
      - 自分でやれば？
      - 中を見てみよう
  - source_sentence: 信用できない
    sentences:
      - どっちでもいいよ
      - 誰？
      - 誰かが呪文で花をぬいぐるみに変えた
  - source_sentence: 例えば？
    sentences:
      - 誰かがが魔法をかけた
      - ジャック
      - なんでしなきゃいけないの？
  - source_sentence: 魔法を使える人
    sentences:
      - かっこいいね
      - 物の姿を変えられる人
      - 町って？
model-index:
  - name: SentenceTransformer based on colorfulscoop/sbert-base-ja
    results:
      - task:
          type: binary-classification
          name: Binary Classification
        dataset:
          name: custom arc semantics data jp
          type: custom-arc-semantics-data-jp
        metrics:
          - type: cosine_accuracy
            value: 0.9044117647058824
            name: Cosine Accuracy
          - type: cosine_accuracy_threshold
            value: 0.5485918521881104
            name: Cosine Accuracy Threshold
          - type: cosine_f1
            value: 0.912751677852349
            name: Cosine F1
          - type: cosine_f1_threshold
            value: 0.47659817337989807
            name: Cosine F1 Threshold
          - type: cosine_precision
            value: 0.918918918918919
            name: Cosine Precision
          - type: cosine_recall
            value: 0.9066666666666666
            name: Cosine Recall
          - type: cosine_ap
            value: 0.9088999169341241
            name: Cosine Ap
          - type: dot_accuracy
            value: 0.9117647058823529
            name: Dot Accuracy
          - type: dot_accuracy_threshold
            value: 293.22845458984375
            name: Dot Accuracy Threshold
          - type: dot_f1
            value: 0.9166666666666666
            name: Dot F1
          - type: dot_f1_threshold
            value: 293.22845458984375
            name: Dot F1 Threshold
          - type: dot_precision
            value: 0.9565217391304348
            name: Dot Precision
          - type: dot_recall
            value: 0.88
            name: Dot Recall
          - type: dot_ap
            value: 0.9171086358892895
            name: Dot Ap
          - type: manhattan_accuracy
            value: 0.9117647058823529
            name: Manhattan Accuracy
          - type: manhattan_accuracy_threshold
            value: 524.0676879882812
            name: Manhattan Accuracy Threshold
          - type: manhattan_f1
            value: 0.918918918918919
            name: Manhattan F1
          - type: manhattan_f1_threshold
            value: 524.0676879882812
            name: Manhattan F1 Threshold
          - type: manhattan_precision
            value: 0.9315068493150684
            name: Manhattan Precision
          - type: manhattan_recall
            value: 0.9066666666666666
            name: Manhattan Recall
          - type: manhattan_ap
            value: 0.9111567321590129
            name: Manhattan Ap
          - type: euclidean_accuracy
            value: 0.9117647058823529
            name: Euclidean Accuracy
          - type: euclidean_accuracy_threshold
            value: 23.82940673828125
            name: Euclidean Accuracy Threshold
          - type: euclidean_f1
            value: 0.918918918918919
            name: Euclidean F1
          - type: euclidean_f1_threshold
            value: 23.82940673828125
            name: Euclidean F1 Threshold
          - type: euclidean_precision
            value: 0.9315068493150684
            name: Euclidean Precision
          - type: euclidean_recall
            value: 0.9066666666666666
            name: Euclidean Recall
          - type: euclidean_ap
            value: 0.9094221163568814
            name: Euclidean Ap
          - type: max_accuracy
            value: 0.9117647058823529
            name: Max Accuracy
          - type: max_accuracy_threshold
            value: 524.0676879882812
            name: Max Accuracy Threshold
          - type: max_f1
            value: 0.918918918918919
            name: Max F1
          - type: max_f1_threshold
            value: 524.0676879882812
            name: Max F1 Threshold
          - type: max_precision
            value: 0.9565217391304348
            name: Max Precision
          - type: max_recall
            value: 0.9066666666666666
            name: Max Recall
          - type: max_ap
            value: 0.9171086358892895
            name: Max Ap

SentenceTransformer based on colorfulscoop/sbert-base-ja

This is a sentence-transformers model finetuned from colorfulscoop/sbert-base-ja on the csv dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: colorfulscoop/sbert-base-ja
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity
  • Training Dataset:
    • csv

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    '魔法を使える人',
    '物の姿を変えられる人',
    'かっこいいね',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Binary Classification

Metric Value
cosine_accuracy 0.9044
cosine_accuracy_threshold 0.5486
cosine_f1 0.9128
cosine_f1_threshold 0.4766
cosine_precision 0.9189
cosine_recall 0.9067
cosine_ap 0.9089
dot_accuracy 0.9118
dot_accuracy_threshold 293.2285
dot_f1 0.9167
dot_f1_threshold 293.2285
dot_precision 0.9565
dot_recall 0.88
dot_ap 0.9171
manhattan_accuracy 0.9118
manhattan_accuracy_threshold 524.0677
manhattan_f1 0.9189
manhattan_f1_threshold 524.0677
manhattan_precision 0.9315
manhattan_recall 0.9067
manhattan_ap 0.9112
euclidean_accuracy 0.9118
euclidean_accuracy_threshold 23.8294
euclidean_f1 0.9189
euclidean_f1_threshold 23.8294
euclidean_precision 0.9315
euclidean_recall 0.9067
euclidean_ap 0.9094
max_accuracy 0.9118
max_accuracy_threshold 524.0677
max_f1 0.9189
max_f1_threshold 524.0677
max_precision 0.9565
max_recall 0.9067
max_ap 0.9171

Training Details

Training Dataset

csv

  • Dataset: csv
  • Size: 680 training samples
  • Columns: text1, text2, and label
  • Approximate statistics based on the first 680 samples:
    text1 text2 label
    type string string int
    details
    • min: 4 tokens
    • mean: 8.29 tokens
    • max: 15 tokens
    • min: 4 tokens
    • mean: 7.97 tokens
    • max: 14 tokens
    • 0: ~40.44%
    • 1: ~59.56%
  • Samples:
    text1 text2 label
    いらない うんよろしく 0
    足元よりも更に深くってどこ？ 足元よりも更に深くってなに？ 1
    他にはないの？ どう思う？ 0
  • Loss: CoSENTLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "pairwise_cos_sim"
}

Evaluation Dataset

csv

  • Dataset: csv
  • Size: 680 evaluation samples
  • Columns: text1, text2, and label
  • Approximate statistics based on the first 680 samples:
    text1 text2 label
    type string string int
    details
    • min: 4 tokens
    • mean: 8.32 tokens
    • max: 15 tokens
    • min: 4 tokens
    • mean: 8.16 tokens
    • max: 14 tokens
    • 0: ~44.85%
    • 1: ~55.15%
  • Samples:
    text1 text2 label
    井戸から水をくんでいた 井戸を使っていた 1
    夕飯は何だったの？ チキンヌードル食べた？ 0
    水を井戸からくんでいた 夜ごはんの前 0
  • Loss: CoSENTLoss with these parameters:
    {
    "scale": 20.0,
    "similarity_fct": "pairwise_cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • learning_rate: 2e-05
  • num_train_epochs: 13
  • warmup_ratio: 0.1
  • fp16: True
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 8
  • per_device_eval_batch_size: 8
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 13
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • eval_use_gather_object: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss custom-arc-semantics-data-jp_max_ap
None 0 - - 0.8596
1.0 68 2.6802 1.7807 0.8872
2.0 136 1.4014 1.7683 0.8945
3.0 204 0.7937 1.9877 0.9039
4.0 272 0.5443 1.9106 0.9075
5.0 340 0.4225 1.9418 0.9109
6.0 408 0.3347 2.0123 0.9107
7.0 476 0.3425 2.0387 0.9094
8.0 544 0.2427 1.9878 0.9103
9.0 612 0.2412 2.0424 0.9178
10.0 680 0.1623 2.0273 0.9188
11.0 748 0.1909 2.0955 0.9220
12.0 816 0.1507 2.2124 0.9157
13.0 884 0.1406 2.2126 0.9171

Framework Versions

  • Python: 3.10.14
  • Sentence Transformers: 3.1.0
  • Transformers: 4.44.2
  • PyTorch: 2.4.1+cu121
  • Accelerate: 0.34.2
  • Datasets: 2.20.0
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CoSENTLoss 

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}