Spaces:
Running
Report for ProsusAI/finbert
Hey Team!🤗✨
We’re thrilled to share some amazing evaluation results that’ll make your day!🎉📊
We have identified 1 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_50agree, split train).
👉Robustness issues (1)
| Vulnerability | Level | Data slice | Metric | Transformation | Deviation |
|---|---|---|---|---|---|
| Robustness | major 🔴 | — | Fail rate = 0.116 | Add typos | 116/1000 tested samples (11.6%) changed prediction after perturbation |
🔍✨Examples
When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 11.6% of the cases. We expected the predictions not to be affected by this transformation.| text | Add typos(text) | Original prediction | Prediction after perturbation | |
|---|---|---|---|---|
| 886 | Stora Enso will record a capital gain of EUR 33 million as a non-recurring financial item in its fourth quarter 2006 results . | Stora Enso will record ac pital gajn of EUR 3 million as a non-recurring financial item in its fourth quarrter 2006 results . | positive (p = 0.88) | neutral (p = 0.87) |
| 633 | The reasons behind the estimate include the rise in 2008 rent levels and several fully-leased office and retail properties , which were completed and added to the company 's investment property portfolio . | Ther easnos behind the estimate include the rkse in 2008 rent levels and several fully-leqsed office and retail lroprrties , which were completed and added to the comoany 's investment prlperty prtfilio . | positive (p = 0.69) | neutral (p = 0.80) |
| 4799 | In February the Elcoteq group sold its St Petersburg facility ; according to unconfirmed information the reason could have been supply problems because of the Russian customs service . | In Vebruary te Elcote qgroup sold its St Peterzburg faxility ; accorring to unconfirmrd information thre reasojn could have been supply problemxs because of th eRuzsian customss ervice . | negative (p = 0.95) | neutral (p = 0.52) |
Checkout out the Giskard Space and improve your model.
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
💡 What's Next?
- The Giskard community is always buzzing with ideas. 🐢🤔 What do you want to see next? Your feedback is our favorite fuel, so drop your thoughts in the community forum! 🗣️💬 Together, we're building something extraordinary.
🙌 Big Thanks!
We're grateful to have you on this adventure with us. 🚀🌟 Here's to more breakthroughs, laughter, and code magic! 🥂✨ Keep hugging that code and spreading the love! 💻 #Giskard #Huggingface #AISafety 🌈👏 Your enthusiasm, feedback, and contributions are what seek. 🌟 Keep being awesome!