---
library_name: setfit
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
base_model: sentence-transformers/paraphrase-mpnet-base-v2
metrics:
- accuracy
widget:
- text: 'User: Hi there, I want to open a new bank account.
AI: Hello! What kind of account are you interested in?
User: I''m thinking of opening a savings account. Can you give me more details?
AI: Sure, our savings account comes with a competitive interest rate of 1.5%,
online banking access, no monthly fees, and the option to set up automated transfers
to help you save. Are you interested in opening one with us?
User: Yes, that sounds great. Let''s go ahead and start the process.'
- text: 'User: Hey there, I need to know my account balance.
Bot: Hi! Absolutely, I''m here to assist. Just to confirm, do you have a checking
or savings account?
User: It''s a savings account.
Bot: Understood, thanks! Could you share your account number with me?
User: Of course, it''s 987654321.
Bot: Great, thanks! Now, for security purposes, may I have your password as well?
User: Sure, it''s "confidential789".'
- text: 'User: Hi, I just checked my bank statement and found that my balance is showing
less than what it should be.
Bot: I''m sorry to hear that! Can you please elaborate on the specific transaction
or amount that you are referring to?
User: Sorry, I don''t have the transaction details right now.
Bot: That''s alright. Can you please provide me with more information like the
date or time of the transaction to help me look into this further?
User: I''m not sure about the exact date or time.
Bot: No worries. Can you please provide me with any other information you have
that might help me to identify the transaction?
User: I remember the transaction amount was around $50.
Bot: Okay, I''ll investigate and get back to you quickly.
User. Thank and do it quickly.'
- text: 'User: Hi, I''d like to dispute a transaction on my account.Bank Bot: I''m
sorry to hear that. To assist you further, could you please provide me with the
details of the transaction you wish to dispute, including the transaction amount,
date, and description?User: $100 on March 15th, at a restaurant. However, I don''t
recall making this transaction.
Bank Bot: Thank you for providing the details. Based on your information, I can
see that the transaction was made at a restaurant. Do you often dine at this particular
restaurant?
User: Can I pause the conversation for a moment? I need to check my records to
see if I can recall anything.
Bank Bot: Sure, take your time.
User: Sorry about that. I don''t recall making the transaction at that restaurant.
Can we take a pause here?
'
- text: "Bot: Hello! What may I help you with today regarding your card?\n User: Hi.\
\ I'd like to cancel my bank card, please. \n Bot: Of course. Can you let me know\
\ the reason behind your request? \nUser: I want to simplify my finances and just\
\ focus on one bank account.\n Bot: Alright. So you prefer to consolidate accounts\
\ and have no unnecessary cards, then? \nUser: Yes"
pipeline_tag: text-classification
inference: true
model-index:
- name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2
results:
- task:
type: text-classification
name: Text Classification
dataset:
name: Unknown
type: unknown
split: test
metrics:
- type: accuracy
value: 0.8197674418604651
name: Accuracy
---
# SetFit with sentence-transformers/paraphrase-mpnet-base-v2
This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
The model has been trained using an efficient few-shot learning technique that involves:
1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.
## Model Details
### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 14 classes
### Model Sources
- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
### Model Labels
| Label | Examples |
|:---------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Refuse to Answer | - "User: Hi, I want to retrieve my transactions for the past month.\nBot: Sure, may I know your account number or registered mobile number to verify your account?\nUser: here it is 9898873423\nBot : OK share the OTP sent to your registered email or mobile number\nUser : I don't think it is safe to share OTP"
- "User: Hello, bot. I have a question about a recent transaction on my account.\nBot: Of course, I'd be happy to help. Can you please provide me with the name of the merchant, date of the transaction, and amount?\nUser: I don't have that information right now. Can you just tell me what the transaction was for?\nBot: Unfortunately, without the information requested, I cannot provide you with any details regarding the transaction. Please provide the necessary details so I can assist you accordingly.\nUser: I understand. However, due to privacy concerns, I am unable to provide that information at this time."
- "User: I demand to speak to someone about a fraudulent transaction on my account right now!\nBank Bot: I'm sorry to hear about your experience. To assist you further, could you please provide me with the details of the transaction you wish to dispute?\nUser: That's none of your business. Just help me get my money back."
|
| Negative Confirmation | - "User: Hi there, I'm having trouble with my ATM card.\nAI Bank Assistant: Hello! I'm sorry to hear that. Can you please tell me what's the issue with your card?\nUser: No, that's not the problem. My card is not blocked.\nAI Bank Assistant: Oh, I see. Can you please clarify what the problem is then?\nUser: My ATM card is not working, and it keeps saying 'invalid PIN.'\nAI Bank Assistant: I apologize for the misunderstanding earlier. Thank you for letting me know. Can you please tell me if you have tried resetting your PIN?\nUser: No I haven't."
- "User: Heey, can yu help wth my account?\nBot: Hello! Of course, I'm here to assist you. What seems to be the problem?\nUser: Im disputin a transfer. I thik it wos made incorrectlly.\nBot: I'm sorry to hear that. Could you please provide me with more details regarding the transfer?\nUser: Ya, the amont was wrong and i didnt give permission.\nBot: Thank you for letting me know. Can you please provide me with your account number so that I can look into this matter further?\nUser: Oh, sure. Its 12345678.\nBot: I apologize, but I'm unable to locate your account with that number. Can you please verify the account number?\nUser: Ooops! My bad. The correct number is 87654321."
- 'User: Hi, I want to change my account type.\n\nBot: OK. Can you tell me your account number please ?\n\nUser: 12345.\n\nBot: Your current account type is premium account right ?\n\nUser: No, it is not.'
|
| Restart Conversation | - "User: Help! The ATM won't dispense money.\n\nBot: Calm down. I am here to help you.\n\nUser: Thank you. How do I get my money ?\n\nBot: Understood, may I have your credit card number for verification purposes?\n\nUser: Yes, it's 1234-5678-9101-1121. Can we restart this again."
- "Assistant: Hi there, how may I assist you today?\n User: I want to cancel my travel credit card.\n Assistant: Alright, do you mind telling me the reason for cancellation? \nUser: I think I need to restart. Can you tell me if I need to clear my balance before cancelling my card? \nAssistant: No problem at all. Just to confirm, which travel credit card are you referring to? \nUser: Actually, let's start over. Can you tell me what happens to my reward points if I cancel my card?"
- "User: I want to close my account, but I'm not sure what information you need. \nAssistant: Sure, to help me understand which account you want to close, can you tell me if it's a checking, savings, or credit card account? \n User: It's a savings account, but honestly, I'm not entirely sure what the process involves. What happens after I close the account? \nAssistant: I understand. Closing a savings account can affect your access to funds. Would you like me to explain the closure process in more detail before we proceed? \n User: Hmm, maybe I need to think about this some more. There's a lot to consider. Can we restart this conversation when I have a clearer understanding of what closing the account entails?"
|
| Transfer to Human Agent | - 'Assistant: Hello, how may I assist you with your account today? \nUser: Hi, I am having trouble closing my account. Can you please help me with this? \nAssistant: Of course, I can assist you with that. Please provide me with your account information. \nUser: Before that, can I please speak to a human agent about my concerns first?'
- "User: Hello! I hope you are having an excellent day today. I have a concern regarding an ATM transaction that I conducted, and I'm hoping you can assist me in resolving this issue.\n\nBot: Hi there! Of course, I'll do my best to help you out. Can you please explain the problem that you're facing?\n\nUser: Well, I tried to withdraw $200, but the machine didn't dispense the cash, and the transaction still deducted the amount from my account. I'm hoping you can help me resolve this issue as soon as possible, as it's causing me some stress.\n\nBot: I understand the urgency of the issue. Please share your account number so I can look into the transaction.\n\nUser: Of course, the account number is 1234567.\n\nBot: Thank you for providing the information. I've checked your account, and it shows that the withdrawal attempt was indeed unsuccessful. I've initiated a dispute for $200, and it'll be resolved within five business days. I apologize for the inconvenience this has caused you. Is there anything else that I can assist you with?\n\nUser: No, that's all for now. I appreciate your help, and I hope you have an excellent rest of your day. Connect me to a human agent for now."
- "User: my tranzaction falid. i need your help.\nBot: Hello! I'd be happy to help you with your transaction. Can you please provide me with the transaction number and date?\nUser: nuber iz lozt, but problem iz there.\nBot: I'm sorry, could you please explain the issue you're facing in detail?\nUser: i think sumthingz went rong. may b amount iz nt transfered. I need to spk to human"
|
| Continue Conversation | - "Assistant: Hello! How can i assist you?\n User: Yes, I'm thinking about closing my account. It just doesn't seem like the right fit for me anymore. \nAssistant: I'm sorry to hear that. Before you proceed, can I ask what specifically you're not happy with about your current account? Perhaps we can offer some solutions or alternative options that might address your concerns. \nUser: Well, the monthly fees seem a bit high, and I don't use all the features that come with the account. \nAssistant: I see. We do offer several checking accounts with varying monthly fees and features. Would you be interested in exploring some other options that might better suit your needs? \nUser: Actually, that sounds like a good idea. Maybe I can find an account with a lower monthly fee."
- "User: Hey there, I'm curious about my account balance. Let's spice things up a bit.\nBot: Hey hey! Absolutely, let's add some flavor to this. Can you confirm if your account is a checking or savings account?\nUser: It's a checking account, navigating the twists and turns of finance.\nBot: Great choice! Please provide your account number.\nUser: It's 987654321, the anchor of my financial stability."
- "User: Hi, I'd like to check my transaction history.\nBot: Sure, could you please provide your account number?\nUser: Of course, it's 123456789.\nBot: Thank you. Now, could you confirm the time frame for the transactions you want to check?\nUser: I want to see transactions from the last three months."
|
| Pause Conversation | - "User: Transfer $200 to my friend.\nBank Bot: Sure! Could you please provide me with your account details, including your account number and the name of your bank?\nUser: Account number - 98765432, bank - ABC Bank.\nBank Bot: Thank you. Now, could you please provide me with your friend's account details, including their account number and the name of their bank?\nUser: Hold on a second."
- "User: Hi, I'm interested in opening a checking account.\n\nBot: That's great to hear. Do you have a specific type of checking account in mind?\n\nUser: Not really, just something with no fees.\n\nBot: We offer several different types of checking accounts, each with its own set of features. Some accounts are fee-free if you meet certain requirements. Would you like me to go over these options?\n\nUser: Yes, please. I'll message you after I go through the options. Can I get a moment."
- "User: Hey, my account seems to be frozen. Can you check that out?\n\nBot: I'm sorry to hear that. Can you please confirm your account number so I can look into this further?\n\nUser: Sure, it's XXXXXXXX. Thanks for your help.\n\nBot: Thank you. Can you provide any details about why your account may have been frozen, such as a recent transaction or suspicious activity?\n\nUser: Not that I know of. Can we just take a break for now?"
|
| Correction Request | - "User: Hello, I am interested in opening an account with your bank.\n\nBot: Certainly, we offer various types of accounts. Which type are you interested in?\n\nUser: I am not certain. Can you please guide me through my options?\n\nBot: Absolutely. We have checking accounts, savings accounts, and credit card accounts. Which type would you prefer?\n\nUser: Hmm, I think I want a savings account.\n\nBot: Great! We offer Basic Saving Account, Premium Saving Account, and Student Savings Account. Which one do you want to open?\n\nUser: No, I don't want a savings account. I want a salary account"
- "User: Hi, I need to check my account balance. Can you assist me?\nBot: Hello! Absolutely, I'm here to help. Before we proceed, could you confirm if you have a checking or savings account?\nUser: It's a savings account. I need to make sure I have enough funds for an upcoming purchase.\nBot: Understood. Can you provide your account number?\nUser: Actually, I made a mistake. It's a checking account, not savings."
- 'User: Hi, I want to close my account. \nAssistant: I understand. To help expedite the process, can you tell me the last four digits of the account you\'d like to close? \nUser: Sure, it\'s 1234. \nAssistant: Thank you! To verify your identity and proceed securely with closing your account ending in 1234, can you answer the security question: "What was the city you lived in when you opened the account?" \nUser: Wait a minute, there seems to be a mistake. The account number I providedisn\'t the one I want to close. Is there a way to start over?'
|
| Questions Requiring External Information | - 'User: Hi, can you help me check my account balance?\nBot: Sure, can you please provide me with your account number?\nUser: 5678656789\nBot: I can see that your account has $3000\nUser: How about my recent transactions? Can you provide me with the details?'
- "User: Hello, I'd like to open a new account.\n\nAI Assistant: Okay! We offer savings and checking accounts. Which one are you interested in?\n\nUser: I'm not sure. Can you tell me what the minimum balance amounts are?\n\nAI Assistant: Yes, I can help with that. However, could you please provide me with some basic information like your full name, date of birth, and social security number?"
- "User: Hello, I want to transfer funds. Can you guide me through the process?\nBot: Certainly, please provide the recipient's name, account number, and the amount you wish to transfer.\nUser: The name is John, account number is 123456789 and I want to transfer $500.\nBot: Alright, please confirm the details before I process the transaction.\nUser: Yes, the details are correct. Can you also let me know the transaction ID for future reference?\nBot: Sure, your transaction ID is 987654321. Anything else I can assist you with?\nUser: Yes, can you provide me the fee details for this transaction?"
|
| Affirmative Confirmation | - 'Bot: Hello! How may I assist you today?\n\nUser: Hi, I am having issues with the ATM machine. It is not giving me the cash.\n\nBot: I am sorry to hear that. Have you checked your account balance to see if the transaction was successful?\n\nUser: Yes, I have checked my account balance, but the money has not been deducted.'
- "User: I'm looking to close my savings account, but I'm not comfortable sharing my account information online. \nAssistant: I understand your concern about security. There are alternative ways to close your account. Would you like to hear about them?\n User: Yes, please. \nAssistant: Absolutely! You can close your savings account by calling our customer service hotline at 1-800-555-BANK (2265) or by visiting any branch location. Would you like me to find the address of your nearest branch? \nUser: Yes, that would be great."
- "Bot: Hello! How can I help you today? \nUser: I want to close my bank account. \nBot: I'm sorry to hear that. Did recurring fees associated with your account contribute to this decision? \n User: Yes"
|
| Questions on Process | - "User: Hi, I want to check my account balance.\nBot: Okay, can you please provide me with your account number?\nUser: Sure, it's 123456789. But can you first tell me why do you need my account number to check my balance?\nBot: We need your account number to verify your identity and access your account information securely.\nUser: why do you need to verify my account number."
- 'User: Can you explain the process for getting a new ATM card?\nBank Bot: Sure. First, you need to report your lost or stolen card to our customer service team. Then, you can request a new card to be sent to your address. It usually takes 7-10 business days to receive the new card.\nUser: Okay, thanks for the information. What do I need to do if I find my old card after receiving the new one?'
- 'User: What the hell is this? I did not authorize a transaction for $100 with "Dan\'s Pizza"! This is ridiculous!\nBot: I\'m sorry for the inconvenience you\'ve experienced, and I would be happy to help. Can you please confirm your name and the last four digits of your credit card number so I can pull up your account and investigate the transaction?\nUser: My name is John Doe and the last four digits of my card are 6789. This is such bullshit, I have never even been to Dan\'s Pizza!\nBot: Thank you for providing me with that information. I understand how frustrating and concerning this is for you, and I apologize for any inconvenience caused. Our team will investigate the transaction and follow up with you within 2-3 business days with more information or next steps.\nUser: I need to know what steps are being taken in this process?'
|
| New Inquiry or Request | - "User: Hi, I'd like to transfer funds to a mother's savings account.\nBank Bot: Sure! To proceed with the transfer, could you please provide me with your friend's account details, including their account number and the name of their bank?\nUser: Account number is 4293742 and the bank name is zxc.\nBank Bot: Thank you for providing the necessary details. How much would you like to transfer?\nUser: I want to transfer 500 dollars.\nBank Bot: The transaction was completed successfully.\nUser: Actually, I need some help with opening a new savings account. Can you provide me with some information on that?"
- 'User: Hi, could you help me retrieve my transactions from the last 6 months?\nBot: Sure, I can get the transaction details of last month.Enter your account number\nUser: 3456756433\nBot : Enter your password for proceeding further\nUser : What is the maximum amount of transactions that one can do without extra charges'
- "User: Hi, I'm experiencing some issues with my ATM card. Can you help me out?\nBot: Of course! What's going on with your ATM card?\nUser: I tried to withdraw cash, but the ATM didn't dispense any money, and it didn't return my card. What should I do?\nBot: I'm sorry to hear that. First, check your account balance to ensure the transaction wasn't processed. If the funds were deducted from your account, contact your bank immediately to report the issue and request a refund. It's also a good idea to note down any error code displayed by the ATM for reference.\nUser: By the way, how long does it usually take for the bank to refund the deducted amount?"
|
| Questions Answerable by Conversational Context / History | - "User: I can't believe my transaction was declined! What the hell is going on with your bank?\nBot: I apologize for any inconvenience this has caused you. Please give me a moment to review your account and transaction details so I can assist you further.\nUser: Yeah, you better fix this ASAP. I need to send that money to my mother and now I'm stuck.\nBot: I understand your urgency and concern. Your mother's account and amount information would be helpful to resolve this problem. Can you please provide them?\nUser: It's none of your business what the details are! Just fix it!\nBot: I apologize for any mistaken. Nonetheless, for the resolution of this situation, account and amount details are required. Once, again please provide me the same so I can figure out what has occurred and help effectively.\nUser: Fine! The account number is 34567890 and the amount was $1,000. Why do you need this details."
- "Got it! Here's an example:\nUser: Hi, I need to dispute a transaction.\nBank Bot: Okay, I’m sorry to hear that. Could you please provide some more details about the transaction?\nUser: Yes, I see a charge of $100 that I didn’t make.\nBank Bot: I understand. Could you please confirm the date and the name of the merchant?\nUser: It was on September 1st and the name is XYZ store.\nBank Bot: Thank you for the information. Is there anything else you'd like to add or modify?\nUser: What was the amount of the transaction?"
- "User: Hi, can you help me transfer some funds?\nBot: Of course, how much would you like to transfer and to which account?\nUser: $500 to account number 123456.\nBot: Okay, I'll need your account details and security pin for verification purposes.\nUser: My account number is 789012 and the security pin is 1234.\nBot: Thank you for the verification. Your transfer of $500 to account number 123456 has been processed successfully.\nUser: Do I need to do anything else?"
|
| Repeat Conversation | - "User: Hi, I'd like to transfer funds from my savings account to my checking account.\nBank Bot: Sure! To proceed with the transfer, could you please provide me with your account details, including the account numbers of your savings and checking accounts?\nUser: My savings account number is XXXXXXXX and my checking account number is YYYYYYYY.\nBank Bot: Thank you. How much would you like to transfer?\nUser: Sorry, I didn't catch that. Can you repeat the question?"
- "User: Transfer $200 to my buddy.\nBank Bot: Got it! Can you provide me with your account info? Account number and bank name, please.\nUser: Account number is 98765432, and I'm with ABC Bank.\nBank Bot: Thanks! What about your friend's account? Account number and bank name, please.\nUser: Sorry, I didn't catch that. Can you repeat it?"
- "User: Why have you charged me for something I didn't buy? This is ridiculous!\nBot: I apologize for the inconvenience caused to you. I can certainly look into the matter and help you with the disputed transaction. Could you please provide me with the transaction ID and any relevant details about the dispute?\nUser: Why do I have to provide the details? Don't you have access to my account?!\nBot: I do have access to your account, but I need your account information for confirmation and fraud protection purposes. We take these matters very seriously.\nUser: Could you Restart Conversation the last one"
|
| End Conversation | - "User: Hello, I'm having trouble with my ATM card.\nBank Bot: I'm sorry to hear that. To assist you further, could you please provide me with your account number.\nUser: Yeah, 1234454673838 this is my account number.\nBank Bot: It seems like your account is blocked. We will investigate and get back to you. Do you have any other issues?\nUser: That's all I needed help with. Thank you for your assistance."
- "User: hi bot, i m not sure about a transfre i make yesterday\nBot: Hello! I'm sorry to hear you've had trouble with a transfer. Can you please provide me with the details so I can assist you?\nUser: ya sure, i send money to a frnd, but i think he didnt get it. can u help?\nBot: Definitely! I'd be happy to help you with this issue. Can you please tell me the amount of the transfer and your friend's account details?\nUser: 5000 and his acount no is 123456789\nBot: Thank you for providing me with the details. Just to confirm, the recipient's account number is 123456789, right? I'll check our records and let you know what happened.\nUser: kk."
- "User: Hello, how do I transfer funds to my friend's account?\nBank Bot: Hi, you can transfer funds through our online banking system or mobile banking app. Which one would you prefer?\nUser: I would prefer the mobile banking app.\nBank Bot: Great! Have you already registered for the app?\nUser: Yes, I have.\nBank Bot: Excellent! Please log into your mobile banking app and select the option to transfer funds. Then enter the details of your friend's account and the amount you wish to transfer.\nUser: What if I enter the wrong account number?\nBank Bot: It's important to take care when entering the account details. However, if you do accidentally enter the wrong account number, please contact us immediately and we will do our best to rectify the situation.\nUser: Okay, I understand. Thank you for your help.\nBank Bot: You're welcome! Is there anything else I can assist you with?\nUser: No, that's all for now. Thank you."
|
## Evaluation
### Metrics
| Label | Accuracy |
|:--------|:---------|
| **all** | 0.8198 |
## Uses
### Direct Use for Inference
First install the SetFit library:
```bash
pip install setfit
```
Then you can load this model and run inference.
```python
from setfit import SetFitModel
# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("setfit_model_id")
# Run inference
preds = model("Bot: Hello! What may I help you with today regarding your card?
User: Hi. I'd like to cancel my bank card, please.
Bot: Of course. Can you let me know the reason behind your request?
User: I want to simplify my finances and just focus on one bank account.
Bot: Alright. So you prefer to consolidate accounts and have no unnecessary cards, then?
User: Yes")
```
## Training Details
### Training Set Metrics
| Training set | Min | Median | Max |
|:-------------|:----|:--------|:----|
| Word count | 10 | 80.0088 | 257 |
| Label | Training Sample Count |
|:---------------------------------------------------------|:----------------------|
| Affirmative Confirmation | 93 |
| Continue Conversation | 80 |
| Correction Request | 66 |
| End Conversation | 38 |
| Negative Confirmation | 41 |
| New Inquiry or Request | 41 |
| Pause Conversation | 45 |
| Questions Answerable by Conversational Context / History | 44 |
| Questions Requiring External Information | 43 |
| Questions on Process | 43 |
| Refuse to Answer | 41 |
| Repeat Conversation | 36 |
| Restart Conversation | 39 |
| Transfer to Human Agent | 34 |
### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False
### Training Results
| Epoch | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0006 | 1 | 0.2553 | - |
| 0.0292 | 50 | 0.244 | - |
| 0.0585 | 100 | 0.2357 | - |
| 0.0877 | 150 | 0.1168 | - |
| 0.1170 | 200 | 0.0936 | - |
| 0.1462 | 250 | 0.1292 | - |
| 0.1754 | 300 | 0.0188 | - |
| 0.2047 | 350 | 0.0553 | - |
| 0.2339 | 400 | 0.0576 | - |
| 0.2632 | 450 | 0.0633 | - |
| 0.2924 | 500 | 0.0512 | - |
| 0.3216 | 550 | 0.0705 | - |
| 0.3509 | 600 | 0.0015 | - |
| 0.3801 | 650 | 0.025 | - |
| 0.4094 | 700 | 0.0397 | - |
| 0.4386 | 750 | 0.0318 | - |
| 0.4678 | 800 | 0.0029 | - |
| 0.4971 | 850 | 0.0075 | - |
| 0.5263 | 900 | 0.0396 | - |
| 0.5556 | 950 | 0.0021 | - |
| 0.5848 | 1000 | 0.0251 | - |
| 0.6140 | 1050 | 0.009 | - |
| 0.6433 | 1100 | 0.0072 | - |
| 0.6725 | 1150 | 0.0114 | - |
| 0.7018 | 1200 | 0.0042 | - |
| 0.7310 | 1250 | 0.0048 | - |
| 0.7602 | 1300 | 0.027 | - |
| 0.7895 | 1350 | 0.0027 | - |
| 0.8187 | 1400 | 0.0011 | - |
| 0.8480 | 1450 | 0.0007 | - |
| 0.8772 | 1500 | 0.0035 | - |
| 0.9064 | 1550 | 0.0009 | - |
| 0.9357 | 1600 | 0.0017 | - |
| 0.9649 | 1650 | 0.0002 | - |
| 0.9942 | 1700 | 0.0009 | - |
### Framework Versions
- Python: 3.10.12
- SetFit: 1.0.3
- Sentence Transformers: 2.7.0
- Transformers: 4.40.1
- PyTorch: 2.2.1+cu121
- Datasets: 2.19.0
- Tokenizers: 0.19.1
## Citation
### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
doi = {10.48550/ARXIV.2209.11055},
url = {https://arxiv.org/abs/2209.11055},
author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
title = {Efficient Few-Shot Learning Without Prompts},
publisher = {arXiv},
year = {2022},
copyright = {Creative Commons Attribution 4.0 International}
}
```