Edit model card

andegpt-embed

This is a sentence-transformers model finetuned from microsoft/mpnet-base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: microsoft/mpnet-base
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 768 tokens
  • Similarity Function: Cosine Similarity
  • Language: es
  • License: apache-2.0

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("enpaiva/embed-andegpt-280724")
# Run inference
sentences = [
    '¿Cuál es el número del artículo que trata sobre la mínima sección permisible para una lámpara o grupo de lámparas?',
    'Reglamento de Baja Tensión de la ANDE: El 14.7.3 trata sobre: La mínima sección permisible para una lámpara, o grupo de lámparas que forman un solo artefacto de iluminación, será de 1 mm².',
    'Reglamento de Baja Tensión de la ANDE: El 19.2.1 trata sobre: La caída de tensión máxima permisible, es la siguiente:  a) Para iluminación, en general (19.1.1), hasta 4%. -2% en el alimentador, y -2% en el circuito (19.1.2). b) Para fuerza motriz y/o calefacción, hasta 5%. -4% en el alimentador, y -1% en el ramal. c) En el caso de clientes que reciban la energía a tensión diferente de las normales de utilización (19.1.3), hasta 4%.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

Metric Value
cosine_accuracy 0.9971
dot_accuracy 0.0032
manhattan_accuracy 0.9968
euclidean_accuracy 0.9971
max_accuracy 0.9971

Training Details

Training Hyperparameters

Non-Default Hyperparameters

  • prediction_loss_only: False
  • learning_rate: 2e-05
  • lr_scheduler_type: cosine
  • log_level_replica: passive
  • log_on_each_node: False
  • logging_nan_inf_filter: False
  • bf16: True
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • prediction_loss_only: False
  • per_device_train_batch_size: 8
  • per_device_eval_batch_size: 8
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • learning_rate: 2e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 3
  • max_steps: -1
  • lr_scheduler_type: cosine
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: passive
  • log_on_each_node: False
  • logging_nan_inf_filter: False
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: True
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss andegpt-dev_max_accuracy
0 0 - - 0.6136
0.0270 250 0.8269 0.3100 0.9658
0.0540 500 0.3667 0.2169 0.9721
0.0809 750 0.2305 0.1594 0.9801
0.1079 1000 0.1866 0.1372 0.9830
0.1349 1250 0.1639 0.1114 0.9859
0.1619 1500 0.1375 0.0983 0.9871
0.1889 1750 0.1082 0.0815 0.9886
0.2158 2000 0.1023 0.0723 0.9900
0.2428 2250 0.0777 0.0703 0.9905
0.2698 2500 0.0809 0.0656 0.9896
0.2968 2750 0.0639 0.0662 0.9891
0.3238 3000 0.0633 0.0590 0.9922
0.3507 3250 0.0545 0.0533 0.9930
0.3777 3500 0.0541 0.0458 0.9932
0.4047 3750 0.0475 0.0365 0.9947
0.4317 4000 0.0394 0.0330 0.9939
0.4587 4250 0.0561 0.0345 0.9939
0.4856 4500 0.0432 0.0327 0.9942
0.5126 4750 0.0417 0.0328 0.9944
0.5396 5000 0.0388 0.0252 0.9949
0.5666 5250 0.033 0.0284 0.9959
0.5936 5500 0.0243 0.0229 0.9964
0.6205 5750 0.023 0.0223 0.9959
0.6475 6000 0.0313 0.0209 0.9966
0.6745 6250 0.0285 0.0208 0.9961
0.7015 6500 0.022 0.0192 0.9961
0.7285 6750 0.0219 0.0235 0.9956
0.7555 7000 0.0258 0.0186 0.9954
0.7824 7250 0.0226 0.0230 0.9959
0.8094 7500 0.0226 0.0240 0.9961
0.8364 7750 0.0208 0.0173 0.9968
0.8634 8000 0.0147 0.0200 0.9956
0.8904 8250 0.0193 0.0147 0.9971
0.9173 8500 0.0254 0.0136 0.9968
0.9443 8750 0.0148 0.0132 0.9971
0.9713 9000 0.0174 0.0157 0.9968
0.9983 9250 0.0221 0.0144 0.9971

Framework Versions

  • Python: 3.11.0
  • Sentence Transformers: 3.0.1
  • Transformers: 4.39.3
  • PyTorch: 2.2.0+cu121
  • Accelerate: 0.28.0
  • Datasets: 2.20.0
  • Tokenizers: 0.15.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
Downloads last month
14
Safetensors
Model size
109M params
Tensor type
F32
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for enpaiva/embed-andegpt-H768

Finetuned
(47)
this model

Evaluation results