metadata

base_model: Snowflake/snowflake-arctic-embed-m
library_name: sentence-transformers
metrics:
  - cosine_accuracy@1
  - cosine_accuracy@3
  - cosine_accuracy@5
  - cosine_accuracy@10
  - cosine_precision@1
  - cosine_precision@3
  - cosine_precision@5
  - cosine_precision@10
  - cosine_recall@1
  - cosine_recall@3
  - cosine_recall@5
  - cosine_recall@10
  - cosine_ndcg@10
  - cosine_mrr@10
  - cosine_map@100
  - dot_accuracy@1
  - dot_accuracy@3
  - dot_accuracy@5
  - dot_accuracy@10
  - dot_precision@1
  - dot_precision@3
  - dot_precision@5
  - dot_precision@10
  - dot_recall@1
  - dot_recall@3
  - dot_recall@5
  - dot_recall@10
  - dot_ndcg@10
  - dot_mrr@10
  - dot_map@100
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:1539
  - loss:MatryoshkaLoss
  - loss:MultipleNegativesRankingLoss
widget:
  - source_sentence: >-
      How do the models ensure the production of valid, reliable, and factually
      accurate outputs while assessing risks associated with content provenance
      and offensive cyber activities?
    sentences:
      - >-
        Information or Capabilities  

        MS-1.1-0 05 Evaluate novel methods and technologies for the measurement
        of GAI-related 

        risks in cluding in  content provenance , oﬀensive cy ber, and CBRN ,
        while 

        maintaining the models’ ability to produce valid, reliable, and
        factually accurate outputs.  Information Integrity ; CBRN 

        Information or Capabilities ; 

        Obscene, Degrading, and/or Abusive Content
      - >-
        Testing. Systems should undergo extensive testing before deployment.
        This testing should follow domain-specific best practices, when
        available, for ensuring the technology will work in its real-world
        context. Such testing should take into account both the specific
        technology used and the roles of any human operators or reviewers who
        impact system outcomes or effectiveness; testing should include both
        automated systems testing and human-led (manual) testing. Testing
        conditions should mirror as
      - >-
        oping technologies related to a sensitive domain and those collecting,
        using, storing, or sharing sensitive data 

        should, whenever appropriate, regularly provide public reports
        describing: any data security lapses or breaches 

        that resulted in sensitive data leaks; the numbe r, type, and outcomes
        of ethical pre-reviews undertaken; a 

        description of any data sold, shared, or made public, and how that data
        was assessed to determine it did not pres-
  - source_sentence: >-
      How should automated systems handle user data in terms of collection and
      user consent according to the provided context?
    sentences:
      - >-
        Property Appraisal and Valuation Equity: Closing the Racial Wealth Gap
        by Addressing Mis-valuations for

        Families and Communities of Color. March 2022.
        https://pave.hud.gov/sites/pave.hud.gov/files/

        documents/PAVEActionPlan.pdf

        53. U.S. Equal Employment Opportunity Commission. The Americans with
        Disabilities Act and the Use of

        Software, Algorithms, and Artificial Intelligence to Assess Job
        Applicants and Employees . EEOC-
      - >-
        defense, substantive or procedural, enforceable at law or in equity by
        any party against the United States, its 

        departments, agencies, or entities, its officers, employees, or agents,
        or any other person, nor does it constitute a 

        waiver of sovereign immunity. 

        Copyright Information 

        This document is a work of the United States Government and is in the
        public domain (see 17 U.S.C. §105). 

        2
      - >-
        privacy through design choices that ensure such protections are included
        by default, including ensuring that data collection conforms to
        reasonable expectations and that only data strictly necessary for the
        specific context is collected. Designers, developers, and deployers of
        automated systems should seek your permission 

        and respect your decisions regarding collection, use, access, transfer,
        and deletion of your data in appropriate
  - source_sentence: >-
      How many participants attended the listening sessions organized for
      members of the public?
    sentences:
      - >-
        37 MS-2.11-0 05 Assess the proportion of synthetic to non -synthetic
        training data and verify 

        training data is not overly homogenous or  GAI-produced to mitigate
        concerns of 

        model collapse.  Harmful Bias and Homogenization  

        AI Actor Tasks:  AI Deployment, AI Impact Assessment, Aﬀected
        Individuals and Communities, Domain Experts, End -Users, 

        Operation and Monitoring, TEVV
      - >-
        lenders who may be avoiding serving communities of color are conducting
        targeted marketing and advertising.51 

        This initiative will draw upon strong partnerships across federal
        agencies, including the Consumer Financial 

        Protection Bureau and prudential regulators. The Action Plan to Advance
        Property Appraisal and Valuation 

        Equity includes a commitment from the agencies that oversee mortgage
        lending to include a
      - >-
        for members of the public. The listening sessions together drew upwards
        of 300 participants. The Science and

        Technology Policy Institute produced a synopsis of both the RFI
        submissions and the feedback at the listeningsessions.

        115

        61
  - source_sentence: >-
      Why is it particularly important to monitor the risks of confabulated
      content when integrating Generative AI (GAI) into applications that
      involve consequential decision making?
    sentences:
      - >-
        of how and what the technologies are doing. Some panelists suggested
        that technology should be used to help people receive benefits, e.g., by
        pushing benefits to those in need and ensuring automated decision-making
        systems are only used to provide a positive outcome; technology
        shouldn't be used to take supports away from people who need them.
      - >-
        many real -world applications, such as in healthcare, where a
        confabulated summary of patient 

        information reports could  cause doctors to make  incorrect diagnoses 
        and/or recommend the wrong 

        treatments.  Risks of confabulated content may be especially important
        to monitor  when integrating GAI 

        into applications involving  consequential  decision making. 

        GAI outputs may also include confabulated logic or citations  that
        purport to justify or explain the
      - >-
        settings or in the public domain.  

        Organizations can restrict AI applications that cause harm, exceed
        stated risk tolerances, or that conﬂict with their tolerances or values.
        Governance tools and protocols that are applied to other types of AI
        systems can be applied to GAI systems. These p lans and actions
        include: 

        • Accessibility and reasonable accommodations  

        • AI actor credentials and qualiﬁcations  

        • Alignment to organizational values  • Auditing and assessment
  - source_sentence: >-
      How does the framework address the concerns related to the rapid
      innovation and changing definitions of AI systems?
    sentences:
      - >-
        or inequality. Assessment could include both qualitative and
        quantitative evaluations of the system. This equity assessment should
        also be considered a core part of the goals of the consultation
        conducted as part of the safety and efficacy review.
      - >-
        deactivate AI systems that demonstrate performance or outcomes
        inconsistent with intended use.  

        Action ID  Suggested Action  GAI Risks  

        MG-2.4-001 Establish and maintain communication plans to inform AI
        stakeholders as part of 

        the deactivation or disengagement process of a speciﬁc GAI system
        (including for open -source  models) or context of use, including r
        easons, workarounds, user 

        access removal, alternative processes, contact information, etc.  Human
        -AI Conﬁguration
      - >-
        SECTION  TITLE

        Applying The Blueprint for an AI Bill of Rights 

        While many of the concerns addressed in this framework derive from the
        use of AI, the technical 

        capabilities and specific definitions of such systems change with the
        speed of innovation, and the potential 

        harms of their use occur even with less technologically sophisticated
        tools. Thus, this framework uses a two-

        part test to determine what systems are in scope. This framework applies
        to (1) automated systems that (2)
model-index:
  - name: SentenceTransformer based on Snowflake/snowflake-arctic-embed-m
    results:
      - task:
          type: information-retrieval
          name: Information Retrieval
        dataset:
          name: Unknown
          type: unknown
        metrics:
          - type: cosine_accuracy@1
            value: 0.9270833333333334
            name: Cosine Accuracy@1
          - type: cosine_accuracy@3
            value: 0.9947916666666666
            name: Cosine Accuracy@3
          - type: cosine_accuracy@5
            value: 1
            name: Cosine Accuracy@5
          - type: cosine_accuracy@10
            value: 1
            name: Cosine Accuracy@10
          - type: cosine_precision@1
            value: 0.9270833333333334
            name: Cosine Precision@1
          - type: cosine_precision@3
            value: 0.33159722222222227
            name: Cosine Precision@3
          - type: cosine_precision@5
            value: 0.19999999999999998
            name: Cosine Precision@5
          - type: cosine_precision@10
            value: 0.09999999999999999
            name: Cosine Precision@10
          - type: cosine_recall@1
            value: 0.9270833333333334
            name: Cosine Recall@1
          - type: cosine_recall@3
            value: 0.9947916666666666
            name: Cosine Recall@3
          - type: cosine_recall@5
            value: 1
            name: Cosine Recall@5
          - type: cosine_recall@10
            value: 1
            name: Cosine Recall@10
          - type: cosine_ndcg@10
            value: 0.969317939271961
            name: Cosine Ndcg@10
          - type: cosine_mrr@10
            value: 0.9587673611111113
            name: Cosine Mrr@10
          - type: cosine_map@100
            value: 0.9587673611111112
            name: Cosine Map@100
          - type: dot_accuracy@1
            value: 0.9270833333333334
            name: Dot Accuracy@1
          - type: dot_accuracy@3
            value: 0.9947916666666666
            name: Dot Accuracy@3
          - type: dot_accuracy@5
            value: 1
            name: Dot Accuracy@5
          - type: dot_accuracy@10
            value: 1
            name: Dot Accuracy@10
          - type: dot_precision@1
            value: 0.9270833333333334
            name: Dot Precision@1
          - type: dot_precision@3
            value: 0.33159722222222227
            name: Dot Precision@3
          - type: dot_precision@5
            value: 0.19999999999999998
            name: Dot Precision@5
          - type: dot_precision@10
            value: 0.09999999999999999
            name: Dot Precision@10
          - type: dot_recall@1
            value: 0.9270833333333334
            name: Dot Recall@1
          - type: dot_recall@3
            value: 0.9947916666666666
            name: Dot Recall@3
          - type: dot_recall@5
            value: 1
            name: Dot Recall@5
          - type: dot_recall@10
            value: 1
            name: Dot Recall@10
          - type: dot_ndcg@10
            value: 0.969317939271961
            name: Dot Ndcg@10
          - type: dot_mrr@10
            value: 0.9587673611111113
            name: Dot Mrr@10
          - type: dot_map@100
            value: 0.9587673611111112
            name: Dot Map@100

SentenceTransformer based on Snowflake/snowflake-arctic-embed-m

This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-m. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Snowflake/snowflake-arctic-embed-m
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 tokens
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Technocoloredgeek/midterm-finetuned-embedding")
# Run inference
sentences = [
    'How does the framework address the concerns related to the rapid innovation and changing definitions of AI systems?',
    'SECTION  TITLE\nApplying The Blueprint for an AI Bill of Rights \nWhile many of the concerns addressed in this framework derive from the use of AI, the technical \ncapabilities and specific definitions of such systems change with the speed of innovation, and the potential \nharms of their use occur even with less technologically sophisticated tools. Thus, this framework uses a two-\npart test to determine what systems are in scope. This framework applies to (1) automated systems that (2)',
    'or inequality. Assessment could include both qualitative and quantitative evaluations of the system. This equity assessment should also be considered a core part of the goals of the consultation conducted as part of the safety and efficacy review.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.9271
cosine_accuracy@3	0.9948
cosine_accuracy@5	1.0
cosine_accuracy@10	1.0
cosine_precision@1	0.9271
cosine_precision@3	0.3316
cosine_precision@5	0.2
cosine_precision@10	0.1
cosine_recall@1	0.9271
cosine_recall@3	0.9948
cosine_recall@5	1.0
cosine_recall@10	1.0
cosine_ndcg@10	0.9693
cosine_mrr@10	0.9588
cosine_map@100	0.9588
dot_accuracy@1	0.9271
dot_accuracy@3	0.9948
dot_accuracy@5	1.0
dot_accuracy@10	1.0
dot_precision@1	0.9271
dot_precision@3	0.3316
dot_precision@5	0.2
dot_precision@10	0.1
dot_recall@1	0.9271
dot_recall@3	0.9948
dot_recall@5	1.0
dot_recall@10	1.0
dot_ndcg@10	0.9693
dot_mrr@10	0.9588
dot_map@100	0.9588

Training Details

Training Dataset

Unnamed Dataset

Size: 1,539 training samples
Columns: sentence_0 and sentence_1
Approximate statistics based on the first 1000 samples:
sentence_0 sentence_1
type string string
details
min: 12 tokens
mean: 23.91 tokens
max: 46 tokens

min: 3 tokens
mean: 84.9 tokens
max: 335 tokens

	sentence_0	sentence_1
type	string	string
details	min: 12 tokens mean: 23.91 tokens max: 46 tokens	min: 3 tokens mean: 84.9 tokens max: 335 tokens

Samples:

sentence_0	sentence_1
`What are confabulations in the context of generative AI outputs, and how do they arise from the design of generative models?`	`Confabulations can occur across GAI outputs and contexts .9,10 Confabulations are a natural result of the way generative models are designed : they generate outputs that approximate the statistical distribution of their training data ; for example, LLMs predict the next token or word in a sentence or phrase . While such statistical prediction can produce factual ly accurate and consistent outputs , it can also produce`
`What roles do Rashida Richardson and Karen Kornbluh hold in relation to technology and democracy as mentioned in the context?`	`products, advanced platforms and services, “Internet of Things” (IoT) devices, and smart city products and services. Welcome : •Rashida Richardson, Senior Policy Advisor for Data and Democracy, White House Office of Science andTechnology Policy •Karen Kornbluh, Senior Fellow and Director of the Digital Innovation and Democracy Initiative, GermanMarshall Fund Moderator :`
`What are some best practices that entities should follow to ensure privacy and security in automated systems?`	`Privacy-preserving security. Entities creating, using, or governing automated systems should follow privacy and security best practices designed to ensure data and metadata do not leak beyond the specific consented use case. Best practices could include using privacy-enhancing cryptography or other types of privacy-enhancing technologies or fine-grained permissions and access control mechanisms, along with conventional system security protocols. 33`

Loss: MatryoshkaLoss with these parameters:

{
    "loss": "MultipleNegativesRankingLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 20
per_device_eval_batch_size: 20
num_train_epochs: 5
multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 20
per_device_eval_batch_size: 20
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1
num_train_epochs: 5
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
eval_use_gather_object: False
batch_sampler: batch_sampler
multi_dataset_batch_sampler: round_robin

Training Logs

Epoch	Step	cosine_map@100
0.6494	50	0.9436
1.0	77	0.9501
1.2987	100	0.9440
1.9481	150	0.9523
2.0	154	0.9488
2.5974	200	0.9549
3.0	231	0.9536
3.2468	250	0.9562
3.8961	300	0.9562
4.0	308	0.9562
4.5455	350	0.9562
5.0	385	0.9588

Framework Versions

Python: 3.10.12
Sentence Transformers: 3.1.1
Transformers: 4.44.2
PyTorch: 2.4.1+cu121
Accelerate: 0.34.2
Datasets: 3.0.0
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}