Netta1994/setfit_baai_wikisum_gpt-4o_improved-cot-instructions_two_reasoning_remove_final_evalu

SetFit with BAAI/bge-base-en-v1.5

This is a SetFit model that can be used for Text Classification. This SetFit model uses BAAI/bge-base-en-v1.5 as the Sentence Transformer embedding model. A LogisticRegression instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a Sentence Transformer with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

Model Details

Model Description

• Model Type: SetFit
• Sentence Transformer body: BAAI/bge-base-en-v1.5
• Classification head: a LogisticRegression instance
• Maximum Sequence Length: 512 tokens
• Number of Classes: 2 classes

Model Sources

• Repository: SetFit on GitHub
• Paper: Efficient Few-Shot Learning Without Prompts
• Blogpost: SetFit: Efficient Few-Shot Learning Without Prompts

Model Labels

Label	Examples
0	'Reasoning why the answer may be good:\n1. The answer covers several important aspects of petting a bearded dragon mentioned in the document, such as using slow movements, using 1 or 2 fingers to stroke the head, and using treats to encourage interaction.\n2. It also mentions key safety practices such as washing hands before and after handling the dragon.\n\nReasoning why the answer may be bad:\n1. The answer includes information ("consistently using a specific perfume or scent...") that is incorrect and not supported by the document.\n2. Some details are omitted from the answer, such as avoiding overhead movements, not petting the dragon when it’s showing signs of stress (like hissing or beard fluffing), and how to handle droopy-eyed dragons properly.\n\nFinal Result:' "Reasoning for the Evaluation:\n\nWhy the answer may be good:\n1. Context Grounding: The answer attempts to provide details on how to identify a funnel spider, which is aligned with the document.\n2. Relevance: The answer focuses on physical characteristics of funnel spiders, which relates to the identification task.\n3. Conciseness: The answer stays on topic and attempts to be straightforward without excessive information.\n\nWhy the answer may be bad:\n1. Context Grounding: The answer provides incorrect details about the funnel spider's appearance, which contradicts the provided document. For instance, it wrongly claims that the spider is light brown or gray and has non-poisonous fangs pointing sideways.\n2. Relevance: Although intended to address the question, the information presented is factually incorrect based on the provided document.\n3. Conciseness: The clarity of the answer is undermined by the inclusion of incorrect descriptions, leading to potential confusion.\n\nFinal Result: \n****" 'The given answer is:\n\n"Luis Figo left Barcelona to join Real Madrid."\n\nReasoning why the answer may be good:\n- None. The answer is completely unrelated to the question asked.\n\nReasoning why the answer may be bad:\n- Context Grounding: The answer is not grounded in the context of the provided document as it does not address the topic of real estate commissions at all.\n- Relevance: The answer does not address the specific question asked, which is about calculating real estate commissions.\n- Conciseness: Although the answer is concise, it is irrelevant and does not provide any related information to the question.\n\nFinal result:'
1	'Reasoning why the answer may be good:\n1. Context Grounding: The answer draws from multiple techniques mentioned in the document, such as quick steady breathing, good posture, engaging stomach muscles, and controlling air release.\n2. Relevance: The answer addresses the specific question of how to hold a note, detailing methods directly related to this objective.\n3. Conciseness: The information is clear, practical, and to the point without diverging into irrelevant or overly detailed explanations.\n\nReasoning why the answer may be bad:\n- Some advice in the answer (e.g., "push out your voice with your sternum") is overly simplified or not explicitly stated in the document.\n- Slight deviation into techniques like "breathe in quickly and steadily throughout the song" could be misinterpreted as contradictory to the document's suggestion of controlled breaths.\n\nFinal Result: ****' 'Reasoning Why the Answer May Be Good:\n1. Context Grounding: The answer uses multiple suggestions directly mentioned in the document, such as journaling, trying new things, and making new friends, thus grounding it in the provided context.\n2. Relevance: The response addresses the question "How to Stop Feeling Empty" by giving actionable methods to combat feelings of emptiness.\n3. Conciseness: The answer is relatively clear and to the point, focusing on specific actions one can take to alleviate empty feelings.\n\nReasoning Why the Answer May Be Bad:\n1. Context Grounding: While the answer does pull from the document, it does not cover the breadth of strategies included in the source text, potentially missing out on some helpful suggestions like adopting a pet or seeking professional help.\n2. Relevance: The answer remains relevant but does not touch on the more in-depth solutions and causes of emptiness mentioned in the document, such as mental health issues or grief, which might be critical for some readers.\n3. Conciseness: The answer is concise but perhaps overly so. It sacrifices depth for brevity, therefore not fully leveraging all the comprehensive advice provided in the document.\n\nFinal Result:' 'Reasoning why the answer may be good:\n1. Context Grounding: The provided answer mentions steps that are in the provided document, such as squeezing out excess water, applying a leave-in conditioner, and detangling with a wide-tooth comb.\n2. Relevance: The answer directly addresses the process of drying curly hair, which is the question asked.\n3. Conciseness: The answer is concise and breaks down the steps succinctly without diving into overly detailed procedures or reasons behind each step.\n\nReasoning why the answer may be bad:\n1. Context Grounding: The answer does not explicitly mention the steps found in the document about using specific conditioners, the use of T-shirts, or the avoidance of towels which were given in detail in the document.\n2. Relevance: While it mentions the general steps, it does not discuss other relevant crucial aspects such as use of anti-frizz and curling products, the method of parting and scrunching hair, and the importance of not touching hair while it dries.\n3. Conciseness: Although concise, the conciseness may bypass significant details that are crucial to comprehensive understanding, such as the reasoning behind each step and the importance ofspecific methods and products.\n\nFinal Result:'

Evaluation

Metrics

Label	Accuracy
all	0.7568

Uses

Direct Use for Inference

First install the SetFit library:

pip install setfit

Then you can load this model and run inference.

from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_wikisum_gpt-4o_improved-cot-instructions_two_reasoning_remove_final_evalu")
# Run inference
preds = model("Reasoning for evaluation:

**Good Points:**
1. **Context Grounding:** The answer accurately describes many of the identifying characteristics of a funnel spider, such as body color, hair coverage, shiny carapace, and large fangs, which are all well-supported and mentioned in the provided document.
2. **Relevance:** The answer directly addresses the question, which is about identifying a funnel spider.

**Bad Points:**
1. **Omissions:** The answer neglects some critical identifying details such as the spider's size, visible spinnerets, gender differences, geographical location (Australia), their hiding spots, the structure of their web, and some behavioral aspects, all of which were documented and could help in identification.
2. **Conciseness:** Although the answer is concise, some important information from the document that would make the identification more comprehensive is missing. 

Final Result: 
**Bad**

The answer, while accurate on the points it covers, is incomplete and misses several key identifying characteristics found in the document.")

Training Details

Training Set Metrics

Training set	Min	Median	Max
Word count	67	151.4225	212

Label	Training Sample Count
0	34
1	37

Training Hyperparameters

• batch_size: (16, 16)
• num_epochs: (1, 1)
• max_steps: -1
• sampling_strategy: oversampling
• num_iterations: 20
• body_learning_rate: (2e-05, 2e-05)
• head_learning_rate: 2e-05
• loss: CosineSimilarityLoss
• distance_metric: cosine_distance
• margin: 0.25
• end_to_end: False
• use_amp: False
• warmup_proportion: 0.1
• l2_weight: 0.01
• seed: 42
• eval_max_steps: -1
• load_best_model_at_end: False

Training Results

Epoch	Step	Training Loss	Validation Loss
0.0056	1	0.2123	-
0.2809	50	0.2521	-
0.5618	100	0.1456	-
0.8427	150	0.0191	-

Framework Versions

• Python: 3.10.14
• SetFit: 1.1.0
• Sentence Transformers: 3.1.0
• Transformers: 4.44.0
• PyTorch: 2.4.1+cu121
• Datasets: 2.19.2
• Tokenizers: 0.19.1

Citation

BibTeX

@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}