metadata

base_model: Snowflake/snowflake-arctic-embed-l
datasets: []
language: []
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:55736
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      Represent this sentence for searching relevant passages: Jan 20 Become a
      Real Life Superhero
    sentences:
      - >
        The army corps is the largest regular army formation, though in wartime
        two or more corps may be combined to form a field army (commanded by a
        general), and field armies in turn may be combined to form an army
        group. 03/28/31
      - >
        01/20 The world is a dangerous place and sometimes there's a need for
        superheroes. Regrettably, there's no real way to gain super strength or
        to fly like in the comic books.
      - |2
         Stone is useful for both building and crafting in The Blockheads. It's easy to get stone, as explained here. 12/09
  - source_sentence: >-
      Represent this sentence for searching relevant passages: today:2046-10-23
      last summer social security benefits paid when?
    sentences:
      - >
        Summer 2045 We pay Social Security benefits monthly. The benefits are
        paid in the month following the month for which they are due. ...
        Generally, the day of the month you receive your benefit payment depends
        on the birth date of the person for whose earnings record you receive
        benefits.
      - >
        07/26/95 Foursquare’s Explore feature is a great tool to help you find
        the best and popular places nearby, including popular restaurants,
        sights, outdoor places, nightlife spots, and shops.
      - >
        Gesso transferring is a way of displaying an image on an alternate
        surface. By using gesso and a reversed image, the image will then take
        on the surface's texture. 04/28/23
  - source_sentence: >-
      Represent this sentence for searching relevant passages: Floor Press
      01/02/91
    sentences:
      - >
        While the stomach flu affects the stomach, influenza only affects the
        respiratory system (nose, throat, and lungs). Symptoms of the flu
        include fever, muscle aches, sore throat, and cough, and influenza can
        be prevented with a flu shot. Thankfully, people with the flu usually
        don't have gastroenteritis symptoms.
      - >
        If you’re looking to build arm and chest strength, the floor press is a
        great exercise to add to your routine. It works like a bench press, but
        by lying directly on the floor, you give your back better support and
        take away your ability to drive with your legs. 2091 Jan 2
      - >
        "Flashes. When the vitreous gel inside your eye rubs or pulls on the
        retina, you may see what looks like flashing lights or lightening
        streaks. You may have experienced this sensation if you have ever been
        hit in the eye and see ""stars."" These flashes of light can appear off
        and on for several weeks or months. 2034 Winter"
  - source_sentence: >-
      Represent this sentence for searching relevant passages: today:2022-11-01
      what to do in west yellowstone in march 49 years ago 
    sentences:
      - >
        ['Yellowstone Zipline Adventure Park. ... ', 'Playmill Theatre. ... ',
        'Raft the Gallatin River. ... ', 'Grizzly and Wolf Discovery Center. ...
        ', 'Cross-Country Ski the Rendezvous Trails. ... ', 'Cowboy Up for a
        Rodeo. ... ', 'Fish Hebgen Lake. ... ', 'Visit Earthquake Lake.']
        03/31/1973
      - >
        12/14/2041 Intelligence in the normal range is a polygenic trait,
        meaning that it is influenced by more than one gene, and in the case of
        intelligence at least 500 genes. Further, explaining the similarity in
        IQ of closely related persons requires careful study because
        environmental factors may be correlated with genetic factors.
      - >
        Are you a student who is having a hard time with algebra? Or perhaps
        you're trying to brush up on your math skills after not using for them
        for years.
  - source_sentence: >-
      Represent this sentence for searching relevant passages: Nov 6 2002 Easter
      seals (philately)
    sentences:
      - >
        "03/08/2050 This is a list of wild forests in the state of New York.
        Lands designated as ""wild forest"" in New York are managed by the New
        York State Department of Environmental Conservation as part of the
        Forest Preserve. Management Wild forests are intended to retain an
        essentially wild and natural character, however management facilitates a
        greater amount of recreational use than areas designated by the state as
        wilderness, which feature an increased sense of remoteness and solitude.
        Most are located within the boundaries of Adirondack Park or Catskill
        Park. List of New York wild forests See also  Albany Pine Bush Long
        Island Central Pine Barrens Rome Sand Plains References External links
        NYS Department of Environmental Conservation: Forest Preserve unit
        descriptions Land units maps: Adirondack Park, Catskill Park wild
        forests wild forests New York wild forests New York wild forests"
      - >
        2017 Winter The Waterfall Model was the first Process Model to be
        introduced. It is also referred to as a linear-sequential life cycle
        model. ... The Waterfall model is the earliest SDLC approach that was
        used for software development. The waterfall Model illustrates the
        software development process in a linear sequential flow.
      - >
        06/11/2002 An Easter seal is a form of charity label issued to raise
        funds for charitable purposes. They are issued by the Easterseals
        charity in the United States, and by the Canadian Easter Seals
        charities. Easter seals are applied to the front of mail to show support
        for particular charitable causes. They are distributed along with
        appeals to donate to the charities they support. Easter seals are a form
        of Cinderella stamp. They do not have any postal value. Cinderella
        stamps

Technical Report and Model Pipeline

To access our technical report and model pipeline scripts visit our github

SentenceTransformer based on Snowflake/snowflake-arctic-embed-l

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Snowflake/snowflake-arctic-embed-l
Maximum Sequence Length: 512 tokens
Output Dimensionality: 1024 tokens
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Represent this sentence for searching relevant passages: Nov 6 2002 Easter seals (philately)',
    '06/11/2002 An Easter seal is a form of charity label issued to raise funds for charitable purposes. They are issued by the Easterseals charity in the United States, and by the Canadian Easter Seals charities. Easter seals are applied to the front of mail to show support for particular charitable causes. They are distributed along with appeals to donate to the charities they support. Easter seals are a form of Cinderella stamp. They do not have any postal value. Cinderella stamps\n',
    '2017 Winter The Waterfall Model was the first Process Model to be introduced. It is also referred to as a linear-sequential life cycle model. ... The Waterfall model is the earliest SDLC approach that was used for software development. The waterfall Model illustrates the software development process in a linear sequential flow.\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

Size: 55,736 training samples
Columns: anchors and positive
Approximate statistics based on the first 1000 samples:
anchors positive
type string string
details
min: 14 tokens
mean: 20.25 tokens
max: 33 tokens

min: 15 tokens
mean: 47.2 tokens
max: 75 tokens

	anchors	positive
type	string	string
details	min: 14 tokens mean: 20.25 tokens max: 33 tokens	min: 15 tokens mean: 47.2 tokens max: 75 tokens

Samples:

anchors	positive
`Represent this sentence for searching relevant passages: are bugs attracted to citronella November 10?`	`Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 11/10`
`Represent this sentence for searching relevant passages: are bugs attracted to citronella 11/10/09?`	`Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 11/10/09`
`Represent this sentence for searching relevant passages: are bugs attracted to citronella Jan 15?`	`Citronella is naturally occurring oil that repels insects. ... “Citronella oil is repellent to mosquitoes to a degree, but the amount being put out by a candle isn't going to be very effective,” Eric Hoffer, president of Hoffer Pest, told TODAY Home. 01/15`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Evaluation Dataset

Unnamed Dataset

Size: 1,000 evaluation samples
Columns: anchors and positive
Approximate statistics based on the first 1000 samples:
anchors positive
type string string
details
min: 12 tokens
mean: 21.64 tokens
max: 40 tokens

min: 20 tokens
mean: 66.86 tokens
max: 512 tokens

	anchors	positive
type	string	string
details	min: 12 tokens mean: 21.64 tokens max: 40 tokens	min: 20 tokens mean: 66.86 tokens max: 512 tokens

Samples:

anchors	positive
`Represent this sentence for searching relevant passages: today:2068-02-10 what is the meaning of the idiom put two and two together last monday`	`put two and two together. to understand something by using the information you have: I didn't tell her George had left, but she noticed his car was gone and put two and two together. (Definition of put two and two together from the Cambridge Academic Content Dictionary © Cambridge University Press) 02/06/2068`
`Represent this sentence for searching relevant passages: Complete the Throat of the World Quest in Skyrim`	`The Throat of the World is the fifth quest in the second act of the Skyrim’s main quest. During this mission, all the mystery about the game’s main antagonist, Alduin, will be revealed to you.`
`Represent this sentence for searching relevant passages: are blanco kitchen faucets good 04/13/86?`	`Nevertheless, these are good to very good faucets built with good quality components throughout, backed by a strong warranty and superior customer service from a well-established company. Blanco sells only kitchen, prep and bar faucets, nothing for the bathroom. Apr 13 1986`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim"
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 32
per_device_eval_batch_size: 32
learning_rate: 1e-06
weight_decay: 0.01
num_train_epochs: 1
warmup_ratio: 0.1
warmup_steps: 400
bf16: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 32
per_device_eval_batch_size: 32
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 1e-06
weight_decay: 0.01
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 400
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
eval_use_gather_object: False
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Click to expand

Epoch	Step	Training Loss	loss
0.0006	1	1.9721	-
0.0057	10	1.9663	-
0.0115	20	1.947	-
0.0172	30	1.9039	-
0.0230	40	1.9672	-
0.0287	50	1.894	-
0.0344	60	1.8953	-
0.0402	70	1.9001	-
0.0459	80	1.8511	-
0.0517	90	1.7816	-
0.0574	100	1.7657	-
0.0631	110	1.6932	-
0.0689	120	1.6445	-
0.0746	130	1.6565	-
0.0804	140	1.5077	-
0.0861	150	1.4675	-
0.0918	160	1.4307	-
0.0976	170	1.2343	-
0.1033	180	1.1075	-
0.1091	190	1.1142	-
0.1148	200	1.0546	0.0897
0.1206	210	0.9872	-
0.1263	220	0.8933	-
0.1320	230	0.8066	-
0.1378	240	0.7317	-
0.1435	250	0.7404	-
0.1493	260	0.6348	-
0.1550	270	0.6399	-
0.1607	280	0.549	-
0.1665	290	0.4844	-
0.1722	300	0.5109	-
0.1780	310	0.4412	-
0.1837	320	0.4451	-
0.1894	330	0.373	-
0.1952	340	0.4318	-
0.2009	350	0.3996	-
0.2067	360	0.3534	-
0.2124	370	0.3795	-
0.2181	380	0.3195	-
0.2239	390	0.313	-
0.2296	400	0.3174	0.1864
0.2354	410	0.3255	-
0.2411	420	0.3172	-
0.2468	430	0.2601	-
0.2526	440	0.2862	-
0.2583	450	0.3042	-
0.2641	460	0.305	-
0.2698	470	0.2722	-
0.2755	480	0.2684	-
0.2813	490	0.2114	-
0.2870	500	0.2599	-
0.2928	510	0.2226	-
0.2985	520	0.213	-
0.3042	530	0.1968	-
0.3100	540	0.2005	-
0.3157	550	0.17	-
0.3215	560	0.2275	-
0.3272	570	0.1482	-
0.3330	580	0.1404	-
0.3387	590	0.1743	-
0.3444	600	0.1887	0.2803
0.3502	610	0.2018	-
0.3559	620	0.18	-
0.3617	630	0.146	-
0.3674	640	0.1308	-
0.3731	650	0.159	-
0.3789	660	0.1528	-
0.3846	670	0.1439	-
0.3904	680	0.1376	-
0.3961	690	0.1451	-
0.4018	700	0.1408	-
0.4076	710	0.1571	-
0.4133	720	0.1318	-
0.4191	730	0.1548	-
0.4248	740	0.1131	-
0.4305	750	0.1171	-
0.4363	760	0.1246	-
0.4420	770	0.1204	-
0.4478	780	0.1418	-
0.4535	790	0.0907	-
0.4592	800	0.1013	0.3217
0.4650	810	0.1067	-
0.4707	820	0.1064	-
0.4765	830	0.1089	-
0.4822	840	0.1044	-
0.4879	850	0.0916	-
0.4937	860	0.1344	-
0.4994	870	0.1377	-
0.5052	880	0.1078	-
0.5109	890	0.0837	-
0.5166	900	0.0893	-
0.5224	910	0.4395	-
0.5281	920	0.6783	-
0.5339	930	0.6341	-
0.5396	940	0.5763	-
0.5454	950	0.5283	-
0.5511	960	0.4955	-
0.5568	970	0.5138	-
0.5626	980	0.4983	-
0.5683	990	0.5239	-
0.5741	1000	0.5368	0.1056
0.5798	1010	0.5011	-
0.5855	1020	0.5244	-
0.5913	1030	0.39	-
0.5970	1040	0.4645	-
0.6028	1050	0.4164	-
0.6085	1060	0.4698	-
0.6142	1070	0.4074	-
0.6200	1080	0.4608	-
0.6257	1090	0.5081	-
0.6315	1100	0.4749	-
0.6372	1110	0.4384	-
0.6429	1120	0.3604	-
0.6487	1130	0.3853	-
0.6544	1140	0.3238	-
0.6602	1150	0.3656	-
0.6659	1160	0.2918	-
0.6716	1170	0.3919	-
0.6774	1180	0.3366	-
0.6831	1190	0.3731	-
0.6889	1200	0.4923	0.0583
0.6946	1210	0.3101	-
0.7003	1220	0.3177	-
0.7061	1230	0.3779	-
0.7118	1240	0.3342	-
0.7176	1250	0.2819	-
0.7233	1260	0.3247	-
0.7290	1270	0.4053	-
0.7348	1280	0.3277	-
0.7405	1290	0.3325	-
0.7463	1300	0.3827	-
0.7520	1310	0.2674	-
0.7577	1320	0.309	-
0.7635	1330	0.3193	-
0.7692	1340	0.3399	-
0.7750	1350	0.4044	-
0.7807	1360	0.3436	-
0.7865	1370	0.851	-
0.7922	1380	0.9553	-
0.7979	1390	0.8694	-
0.8037	1400	0.8736	0.0333
0.8094	1410	0.7984	-
0.8152	1420	0.8228	-
0.8209	1430	0.8026	-
0.8266	1440	0.8568	-
0.8324	1450	0.8529	-
0.8381	1460	0.757	-
0.8439	1470	0.779	-
0.8496	1480	0.8002	-
0.8553	1490	0.8532	-
0.8611	1500	0.7195	-
0.8668	1510	0.7598	-
0.8726	1520	0.8295	-
0.8783	1530	0.7588	-
0.8840	1540	0.7698	-
0.8898	1550	0.792	-
0.8955	1560	0.8175	-
0.9013	1570	0.7195	-
0.9070	1580	0.7383	-
0.9127	1590	0.4577	-
0.9185	1600	0.0621	0.0207
0.9242	1610	0.0644	-
0.9300	1620	0.0578	-
0.9357	1630	0.0368	-
0.9414	1640	0.056	-
0.9472	1650	0.059	-
0.9529	1660	0.0442	-
0.9587	1670	0.0527	-
0.9644	1680	0.0651	-
0.9701	1690	0.0515	-
0.9759	1700	0.0512	-
0.9816	1710	0.0543	-
0.9874	1720	0.0676	-
0.9931	1730	0.0593	-
0.9989	1740	0.0558	-

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.0.1
Transformers: 4.43.4
PyTorch: 2.4.0+cu121
Accelerate: 0.33.0
Datasets: 2.20.0
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}