SepKeyPro/bge-base-en-trivia-anchor-positive

bge base trained on trivia anchor-positive

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("SepKeyPro/bge-base-en-trivia-anchor-positive")
# Run inference
sentences = [
    'The Azores island group is administered by which country?',
    'The Azores Islands - Portugal | Portugal.com Porto and the North Azores Consisting of nine islands, the Azores \xa0are divided into three groups: the eastern ( Sao Miguel and Santa\xa0Maria islands), the central ( Terceira , Graciosa , Sao Jorge ,\xa0 Pico and Faial islands), and the western ( Flores and Corvo \xa0islands). Apart from international airports of Santa Maria,\xa0Ponta Delgada and Angra do Heroismo, there are flights to the islands\xa0(operated by the regional airline TAP Air Portugal) and ferry boats\xa0between the islands. Even the blase visitor will be touched by the sapphire blue and emerald green lakes, fertile prairies, volcanic cones and craters, colorful hydrangeas and azaleas, 15th century churches, and majestic manor houses. This legendary land, consisting of nine poetically-named islands, enjoys year-round mild temperatures (between 14°C and 22°C–57°F and 71°F) and is a peaceful shelter with a population of 250000 inhabitants, for whom the words “stress” and “pollution” are unheard. There are many stories to tell of the archipelago’s beauty, of fishermen or shepherds, but among them there is one which was told by a holidaymaker. As a foreign couple was silently looking at the Caldeira das Sete Cidades when they were interrupted by their six-year-old son, who asked them: “Is this God’s home?” Sao Miguel Island The largest of all. In Ponta Delgada, the capital, the famous 18th century portals open up to a number of monuments that are worth visiting, most of them built between the 16th and the 18th century: Carlos Machado Museum and churches of Sao Sebastiao, Sao Pedro, Sao Jose, Colegio and Nossa Senhora da Conceicao; convent and chapel of Nossa Senhora da Esperanca and Santa Ana Chapel. Palaces: Fonte Bela and Santa Ana; Conceicao and Santa Catarina; Casa de Carlos Bicudo and the Pacos do Concelho. Other places to visit: Caldeira das Sete Cidades (green and blue lakes); Lagoa do Fogo; Ribeira Grande; Vale das Furnas (spas and hot mineral pools) and Vila Franca do Campo.\xa0 Terceira Island The historic centre of its capital, Angra do Heroismo, has been classified in UNESCO’s International Heritage list. Special reference to the forts of Sao Sebastiao and Sao Joao Baptista (16th-17th-centuries); the palaces of the Bettencourts (Baroque) and of the Capitaes-Generais; the Cathedral, with its silver altar front and treasure; the churches of Colegio dos Jesuitas, Sao Goncalo and Nossa Senhora da Conceicao (17th-century); the churches of Misericordia and Nossa Senhora da Guia (18th-century, the latter encloses the Angra Museum). Other points of interest: Praia da Vitoria, Santa Barbara, Sao Sebastiao and Vila Nova. Graciosa Island In Santa Cruz da Graciosa you will find ancient streets and manor-houses, a beautiful mother-church (16th-18th centuries), Santo Cristo Church (16th century), Cruz da Barra (Manueline) and Ethnographic House. In the Furna do Enxofre, dazzling sights and a vaulted cave over an underground lake (between 11am and 2pm the sunlight filters in). You must also visit Guadalupe and its Baroque church, Luz and Praia (typical windmills). Faial Island In Horta, a famous yacht harbor, look at the beautiful tiles and gilded carvings in the 17th and 18th century churches of Sao Salvador, Nossa Senhora do Carmo and Sao Francisco. To visit: Sacred Art Museum, Nossa Senhora das Angústias Church, Nossa Senhora do Pilar Chapel, Imperio dos Nobres and Porto Pim fortifications, Caldeira Natural Reserve, Capelinhos, grottoes and caves in Costa da Feteira and Monte da Guia belvedere. Pico Island Owes its name to the 7713 ft high volcanic cone. Special reference to Sao Roque do Pico, with its 18th century churches of Sao Roque and Sao Pedro de Alcântara; Lajes do Pico, with its Whale Museum; Madalena, with its Wine Museum and 17th-century church, and Areia Larga, with beautiful winery manor houses. Other places: Calheta de Nesquim, Candelaria, Criacao Velha, Piedade (forest preserve), Prainha do Norte, Santa Luzia, Santo Amaro, Sao Caetano, Sao Joao and Sao Mateus. Sao Jorge Island Velas, with its fishing port, is the main to',
    'Football - Summer Olympic Sport Football Singapore 2010 adopts new sport formats 12 Aug 2010 Football has its roots in ancient China, while the modern version of the game began on the streets of medieval England before evolving into the most popular sport in the world. Medieval origins Modern football has its origins in the streets of medieval England. Neighbouring towns would play each other in games where a heaving mass of players would struggle to drag a pig’s bladder by any means possible to markers at either end of town. A royal ban Football became so violent in England it was banned by the king for more than 300 years. English public schools are credited with subsequently establishing the modern football codes, thus turning the mob riot into a sport in the 16th century. Olympic history Football first appeared on the programme of the Games of the II Olympiad, Paris 1900. It has been on the programme of each edition of the Games ever since, with the exception of Los Angeles 1932. Europe dominated the competition until after 1992 in Barcelona, where Spain became the last European team to win a gold medal. Since the 1996 Olympic Games in Atlanta, African and South American teams have won all the gold medals. Also in 1996, women’s football was introduced into the Olympic programme. Three times, the USA has been on the highest step of the podium - in 1996, in 2004 in Athens and in 2008 in Beijing. But this team was beaten by the Norwegians in the final of the 2000 Games in Sydney.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

• Dataset: trivia-anchor-positive-dev
• Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.672
cosine_accuracy@3	0.842
cosine_accuracy@5	0.877
cosine_accuracy@10	0.914
cosine_precision@1	0.672
cosine_precision@3	0.2807
cosine_precision@5	0.1754
cosine_precision@10	0.0914
cosine_recall@1	0.672
cosine_recall@3	0.842
cosine_recall@5	0.877
cosine_recall@10	0.914
cosine_ndcg@10	0.8005
cosine_mrr@10	0.7634
cosine_map@100	0.7662
dot_accuracy@1	0.672
dot_accuracy@3	0.842
dot_accuracy@5	0.877
dot_accuracy@10	0.914
dot_precision@1	0.672
dot_precision@3	0.2807
dot_precision@5	0.1754
dot_precision@10	0.0914
dot_recall@1	0.672
dot_recall@3	0.842
dot_recall@5	0.877
dot_recall@10	0.914
dot_ndcg@10	0.8005
dot_mrr@10	0.7634
dot_map@100	0.7662

Training Details

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• gradient_accumulation_steps: 4
• learning_rate: 2e-05
• num_train_epochs: 1
• lr_scheduler_type: cosine
• warmup_ratio: 0.1
• bf16: True
• tf32: True
• optim: adamw_torch_fused
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 4
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: cosine
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: True
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss	trivia-anchor-positive-dev_cosine_map@100
0	0	-	-	0.7809
0.0710	10	0.1474	-	-
0.1421	20	0.1112	-	-
0.2131	30	0.0828	-	-
0.2842	40	0.0767	-	-
0.3552	50	0.0575	-	-
0.4263	60	0.0614	-	-
0.4973	70	0.0542	-	-
0.5684	80	0.0566	-	-
0.6394	90	0.068	-	-
0.7105	100	0.072	-	-
0.7815	110	0.0872	-	-
0.8526	120	0.0654	-	-
0.9236	130	0.0793	-	-
0.9947	140	0.0563	-	-
0.0710	10	0.0222	-	-
0.1421	20	0.0096	-	-
0.2131	30	0.0093	-	-
0.2842	40	0.0106	-	-
0.3552	50	0.0078	-	-
0.4263	60	0.0099	-	-
0.4973	70	0.01	-	-
0.5684	80	0.0166	-	-
0.6394	90	0.0272	-	-
0.7105	100	0.041	-	-
0.7815	110	0.0677	-	-
0.8526	120	0.0539	-	-
0.9236	130	0.074	-	-
0.9947	140	0.0484	-	0.7792
0.0710	10	0.0028	-	-
0.1421	20	0.0026	-	-
0.2131	30	0.0021	-	-
0.2842	40	0.0075	-	-
0.3552	50	0.0021	-	-
0.4263	60	0.0026	-	-
0.4973	70	0.0028	-	-
0.5684	80	0.004	-	-
0.6394	90	0.006	-	-
0.7105	100	0.0137	-	-
0.7815	110	0.0449	-	-
0.8526	120	0.0433	-	-
0.9236	130	0.0693	-	-
0.9947	140	0.0451	0.0405	0.7751
0.0710	10	0.0009	-	-
0.1421	20	0.0022	-	-
0.2131	30	0.0007	-	-
0.2842	40	0.001	-	-
0.3552	50	0.0009	-	-
0.4263	60	0.0009	-	-
0.4973	70	0.0011	-	-
0.5684	80	0.0015	-	-
0.6394	90	0.0019	-	-
0.7105	100	0.0037	-	-
0.7815	110	0.0229	-	-
0.8526	120	0.0318	-	-
0.9236	130	0.0661	-	-
0.9947	140	0.0451	-	0.7662

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 3.0.1
• Transformers: 4.42.1
• PyTorch: 2.3.0+cu121
• Accelerate: 0.31.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}