README.md

metadata

base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
  - accuracy
pipeline_tag: text-classification
tags:
  - setfit
  - sentence-transformers
  - text-classification
  - generated_from_setfit_trainer
widget:
  - text: >-
      Reasoning:


      **Context Grounding:**

      - The answer accurately pulls information directly from the provided
      document, including specific changes Haribabu Kommi is making to the
      storage AM. It lists the changes in a manner that seems consistent with
      the details given in the document.


      **Relevance:**

      - The answer is directly relevant to the question, which asked
      specifically about the changes Haribabu Kommi is making to the storage AM.
      It enumerates the exact modifications and additions that are being
      incorporated based on the email content.


      **Conciseness:**

      - The answer is concise and to the point, listing only the changes
      mentioned in the supplied email without deviating into unrelated topics or
      providing extraneous information.


      Final Result: Good
  - text: >-
      **Good**


      **Reasoning:**

      1. **Context Grounding:** The answer "China's Ning Zhongyan won the gold
      medal in the men's 1,500m final at the speed skating World Cup" is
      well-supported by the provided document, which explicitly states that Ning
      Zhongyan won the gold medal in the men's 1,500m final.

      2. **Relevance:** The answer directly addresses the specific question
      asked, identifying the athlete who won the gold medal in the men's 1,500m
      final.

      3. **Conciseness:** The answer is clear and to the point, providing only
      the necessary information without any additional, unrelated details.
  - text: >-
      Reasoning why the answer may be good:

      1. **Context Grounding:** The details in the answer about the sizes of the
      individual and combined portraits are directly pulled from the provided
      document.

      2. **Relevance:** The answer strictly addresses the question about the
      available sizes for the individual and combined portraits without
      deviating into unrelated topics.

      3. **Conciseness:** The answer is concise, directly providing the
      requested size information without including extraneous details.


      Reasoning why the answer may be bad:

      1. There is no discernible reason why this answer may be bad based on the
      provided criteria. It is well-supported by the document, directly answers
      the question, and is concise.


      Final Result: **Good**
  - text: >-
      Reasoning why the answer may be good:

      1. **Context Grounding:** The answer accurately lists the components found
      in the provided document, such as comprehension questions, writing
      exercises, discussion questions, an additional reading list, semester and
      full-year schedules, and a bibliography. It also includes details about
      the organization of the guide into units and lessons, which is mentioned
      in the document.

      2. **Relevance:** The answer specifically addresses the question by
      identifying the components of the British Medieval Student Guide.

      3. **Conciseness:** The answer is relatively to the point, mentioning the
      main components without unnecessary elaboration.


      Reasoning why the answer may be bad:

      1. **Context Grounding:** Although the details are generally correct, some
      parts of the provided description are omitted, such as the note that
      comprehension question answers are in the Teacher's Guide.

      2. **Relevance:** The initial part about the introductory question "Why
      read great literature?" and some other additional comments are not
      directly related to the components of the Student Guide.

      3. **Conciseness:** The answer could be more concise by excluding repeated
      and unrelated information, focusing only on listing the components
      directly.


      Final Result: **Bad**


      The answer introduces unnecessary elements that are not related to
      enumerating the components of the guide, and it overlooks some specific
      details provided in the document. Overall, the response is correct but not
      optimal in addressing the specific question concisely.
  - text: >-
      **Reasoning:**


      **Why the answer may be good:**

      1. It lists three names of Members of Congress, which directly responds to
      the question.


      **Why the answer may be bad:**

      1. **Context Grounding:** The provided document specifically names Rep.
      Danny Davis as the third Member of Congress, and the first two were Reps.
      Keith Ellison and Barbara Lee, not Andy Harris, Kyle Evans, or Jessica
      Smith. This indicates that the actual names provided in the answer are
      incorrect and not grounded in the given context.

      2. **Relevance:** The answer is irrelevant because it provides incorrect
      names, which does not address the question accurately.

      3. **Conciseness:** The answer is concise, but since it’s incorrect, its
      brevity doesn't contribute to its correctness.


      **Final Result:** Bad
inference: true
model-index:
  - name: SetFit with BAAI/bge-base-en-v1.5
    results:
      - task:
          type: text-classification
          name: Text Classification
        dataset:
          name: Unknown
          type: unknown
          split: test
        metrics:
          - type: accuracy
            value: 0.92
            name: Accuracy

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: BAAI/bge-base-en-v1.5
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	"Reasoning:\n\nWhy the Answer May Be Good:\n1. Context Grounding: The answer references the points made in the document, such as Coach Brian Shaw's strategy of pushing the ball after makes and misses as well as encouraging players to take the first available shot within the rhythm of the offense.\n2. Relevance: The answer directly addresses why the Nuggets are having an offensive outburst, highlighting the coaching strategy and players' adaptation.\n3. Conciseness: The answer is mostly to the point and focuses on the main question.\n\nWhy the Answer May Be Bad:\n1. Context Grounding: The mention of a new training technique using virtual reality is not supported by any information within the document provided.\n2. Conciseness: The additional detail about the virtual reality training is unnecessary given that it is not referenced in the document and does not contribute to answering the specific question about the offensive outburst.\n \nFinal Result:\nBased on the evaluation criteria, the inclusion of fictitious or unsupported information about the virtual reality training significantly detracts from the answer’s credibility and relevance.\n\nBad" 'Reasoning why the answer may be good:\n1. Context Grounding: The provided answer cites specific information about film and digital photography directly from the provided document, showing a good grounding.\n2. Relevance: The answer addresses the specific question by discussing different aspects such as exposure tolerance, color capture, and overall image resolution between film and digital photography.\n3. Conciseness: The answer is relatively concise and sticks to the main points relevant to the question without unnecessary elaboration.\n\nReasoning why the answer may be bad:\n1. Overly Detailed: The answer could be seen as too detailed in certain segments, which might slightly detract from conciseness.\n2. Possible Confusion: The mention of specific technical details like "5MP digital sensors" could confuse readers who are not familiar with the technical specifications, detracting from clarity.\n3. Omission of Key Comparison Points: The answer does not touch upon some of the more subjective observations made by the author, like the practical advantages in using film for certain types of photography.\n\nFinal Result: Good' 'Reasoning:\n1. Context Grounding: The answer provided does not reference the third book of the Arcana Chronicles by Kresley Cole or even discuss any content relevant to it. Instead, it discusses an MMA event in Calgary, Alberta, Canada.\n2. Relevance: The answer is entirely irrelevant to the question. The question is about the main conflict in the third book of a specific book series, but the answer describes an MMA fight event.\n3. Conciseness: While the answer is concise in its context, it is entirely off-topic and therefore does not satisfy the conciseness criterion in a meaningful way.\n\nThe answer may be deemed bad because it does not address the question about the Arcana Chronicles at all and instead provides unrelated information about an MMA event.\n\nFinal result: Bad'
1	'Reasoning:\n\n1. Context Grounding:\n - Good: The answer is supported by the document. The suggestions mentioned (getting to know the client, signing a contract, and showcasing honesty and diplomacy) are directly referenced in the text provided.\n - Bad: There is no significant bad aspect in terms of context grounding; the answer sticks closely to the source material.\n\n2. Relevance:\n - Good: The answer is highly relevant to the question about best practices to avoid unnecessary revisions and conflicts. It addresses client understanding, contractual agreements, and the handling of extra charges—all crucial for minimizing conflicts.\n - Bad: There is no deviation from the topic. The answer is focused solely on the best practices, as asked in the question.\n\n3. Conciseness:\n - Good: The answer is concise and to the point, effectively summarizing the practices without unnecessary details.\n - Bad: The level of detail might be too succinct for some readers looking for more in-depth discussion, but this is minor given the criteria.\n\nFinal Result:\nGood' "Reasoning for why the answer may be good:\n- The answer references the author’s emphasis on drawing from personal experiences of pain and emotion to create genuine and relatable characters, which is well-supported by the document.\n- It highlights the importance of genuineness and relatability, which aligns directly with the content provided in the document.\n- The answer stays focused on the specific question about creating a connection between the reader and the characters.\n\nReasoning for why the answer may be bad:\n- The answer could be seen as slightly verbose and might include more detail than necessary, rather than being extremely concise.\n- It does not explicitly mention the document's use of pain for romance authors specifically, which might add to the context.\n\nFinal result: Good" "Reasoning:\n\nPros:\n1. Context Grounding: The document explicitly states that Mauro Rubin is the CEO of JoinPad and mentions that he was speaking at the event, which directly supports the answer.\n2. Relevance: The answer directly and correctly responds to the question about the CEO's identity during the event.\n3. Conciseness: The answer is brief and to the point, providing only the necessary information.\n\nCons:\n- There are no significant cons as the answer fulfills all criteria effectively.\n\nFinal Result: Good"

Evaluation

Metrics

Label	Accuracy
all	0.92

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_rag_ds_gpt-4o_improved-cot-instructions_two_reasoning_only_reasoning_1726")
# Run inference
preds = model("**Good**

**Reasoning:**
1. **Context Grounding:** The answer \"China's Ning Zhongyan won the gold medal in the men's 1,500m final at the speed skating World Cup\" is well-supported by the provided document, which explicitly states that Ning Zhongyan won the gold medal in the men's 1,500m final.
2. **Relevance:** The answer directly addresses the specific question asked, identifying the athlete who won the gold medal in the men's 1,500m final.
3. **Conciseness:** The answer is clear and to the point, providing only the necessary information without any additional, unrelated details.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	52	125.5070	199

Label	Training Sample Count
0	34
1	37

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (5, 5)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0056	1	0.2031	-
0.2809	50	0.2589	-
0.5618	100	0.2125	-
0.8427	150	0.0079	-
1.1236	200	0.0022	-
1.4045	250	0.0017	-
1.6854	300	0.0017	-
1.9663	350	0.0014	-
2.2472	400	0.0014	-
2.5281	450	0.0012	-
2.8090	500	0.0012	-
3.0899	550	0.0012	-
3.3708	600	0.0012	-
3.6517	650	0.0011	-
3.9326	700	0.0011	-
4.2135	750	0.0011	-
4.4944	800	0.0011	-
4.7753	850	0.001	-

Framework Versions

• Python: 3.10.14
• SetFit: 1.1.0
• Sentence Transformers: 3.1.0
• Transformers: 4.44.0
• PyTorch: 2.4.1+cu121
• Datasets: 2.19.2
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}