Hi Team,
This is a report from Giskard Bot Scan 🐢.
We have identified 11 potential vulnerabilities in your model based on an automated scan.
This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).
👉Performance issues (9)
For records in the dataset where text contains "mn", the Precision is 44.76% lower than the global Precision.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "mn" |
Precision = 0.406 |
-44.76% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 8 |
Foundries division reports its sales increased by 9.7 % to EUR 63.1 mn from EUR 57.5 mn in the corresponding period in 2006 , and sales of the Machine Shop division increased by 16.4 % to EUR 41.2 mn from EUR 35.4 mn in the corresponding period in 2006 . |
positive |
neutral (p = 0.51) |
| 12 |
Net income from life insurance doubled to EUR 6.8 mn from EUR 3.2 mn , and net income from non-life insurance rose to EUR 5.2 mn from EUR 1.5 mn in the corresponding period in 2009 . |
positive |
neutral (p = 0.47) |
| 27 |
Net income from life insurance rose to EUR 16.5 mn from EUR 14.0 mn , and net income from non-life insurance to EUR 22.6 mn from EUR 15.2 mn in 2009 . |
positive |
neutral (p = 0.50) |
For records in the dataset where text contains "quarter", the Precision is 36.95% lower than the global Precision.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "quarter" |
Precision = 0.463 |
-36.95% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 1 |
For the last quarter of 2010 , Componenta 's net sales doubled to EUR131m from EUR76m for the same period a year earlier , while it moved to a zero pre-tax profit from a pre-tax loss of EUR7m . |
positive |
neutral (p = 0.50) |
| 206 |
The transactions would increase earnings per share in the first quarter by some EUR0 .28 . |
positive |
neutral (p = 0.55) |
| 247 |
Ruukki 's order book at the end of 2010 was 30 % up year-on-year and 10 % up quarter-on-quarter . |
positive |
neutral (p = 0.80) |
For records in the dataset where text contains "period", the Precision is 31.58% lower than the global Precision.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "period" |
Precision = 0.503 |
-31.58% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 1 |
For the last quarter of 2010 , Componenta 's net sales doubled to EUR131m from EUR76m for the same period a year earlier , while it moved to a zero pre-tax profit from a pre-tax loss of EUR7m . |
positive |
neutral (p = 0.50) |
| 8 |
Foundries division reports its sales increased by 9.7 % to EUR 63.1 mn from EUR 57.5 mn in the corresponding period in 2006 , and sales of the Machine Shop division increased by 16.4 % to EUR 41.2 mn from EUR 35.4 mn in the corresponding period in 2006 . |
positive |
neutral (p = 0.51) |
| 12 |
Net income from life insurance doubled to EUR 6.8 mn from EUR 3.2 mn , and net income from non-life insurance rose to EUR 5.2 mn from EUR 1.5 mn in the corresponding period in 2009 . |
positive |
neutral (p = 0.47) |
For records in the dataset where text contains "eur", the Precision is 31.05% lower than the global Precision.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "eur" |
Precision = 0.507 |
-31.05% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 8 |
Foundries division reports its sales increased by 9.7 % to EUR 63.1 mn from EUR 57.5 mn in the corresponding period in 2006 , and sales of the Machine Shop division increased by 16.4 % to EUR 41.2 mn from EUR 35.4 mn in the corresponding period in 2006 . |
positive |
neutral (p = 0.51) |
| 12 |
Net income from life insurance doubled to EUR 6.8 mn from EUR 3.2 mn , and net income from non-life insurance rose to EUR 5.2 mn from EUR 1.5 mn in the corresponding period in 2009 . |
positive |
neutral (p = 0.47) |
| 27 |
Net income from life insurance rose to EUR 16.5 mn from EUR 14.0 mn , and net income from non-life insurance to EUR 22.6 mn from EUR 15.2 mn in 2009 . |
positive |
neutral (p = 0.50) |
For records in the dataset where text contains "year", the Precision is 27.9% lower than the global Precision.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "year" |
Precision = 0.530 |
-27.90% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 1 |
For the last quarter of 2010 , Componenta 's net sales doubled to EUR131m from EUR76m for the same period a year earlier , while it moved to a zero pre-tax profit from a pre-tax loss of EUR7m . |
positive |
neutral (p = 0.50) |
| 31 |
The company 's order book stood at 1.5 bln euro $ 2.2 bln on September 30 , 2007 , up by 24.2 pct on the year , with international orders amounting to 365 mln euro $ 534.3 mln . |
positive |
neutral (p = 0.53) |
| 141 |
Its market share widened to 48.51 percent from 48.31 percent a year earlier . |
positive |
neutral (p = 0.65) |
For records in the dataset where text contains "operating", the Precision is 21.37% lower than the global Precision.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "operating" |
Precision = 0.578 |
-21.37% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 68 |
Operating profit was EUR 24.5 mn , up from EUR 23.0 mn . |
positive |
neutral (p = 0.49) |
| 123 |
Finnish silicon wafer technology company Okmetic Oyj ( OMX Helsinki : OKM1V ) reported on Thursday ( 7 August ) an operating profit of EUR5 .3 m for the period January-June 2008 , up from EUR3 .3 m in the corresponding period in 2007 . |
positive |
neutral (p = 0.47) |
| 173 |
Operating profit was EUR 11.4 mn , up from EUR 7.5 mn . |
positive |
neutral (p = 0.47) |
For records in the dataset where text contains "share", the Balanced Accuracy is 12.99% lower than the global Balanced Accuracy.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "share" |
Balanced Accuracy = 0.510 |
-12.99% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 10 |
Its board of directors will propose a dividend of EUR0 .12 per share for 2010 , up from the EUR0 .08 per share paid in 2009 . |
positive |
neutral (p = 0.56) |
| 141 |
Its market share widened to 48.51 percent from 48.31 percent a year earlier . |
positive |
neutral (p = 0.65) |
| 201 |
The Finnish government announced Wednesday that it sold a 32 percent stake in chemicals and fertilizer group Kemira Oyj for ( x20ac ) 655.6 million ( $ 890US million ) , sending the company 's share price up 6 percent . |
positive |
neutral (p = 0.75) |
For records in the dataset where text contains "million", the Precision is 11.33% lower than the global Precision.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "million" |
Precision = 0.652 |
-11.33% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 201 |
The Finnish government announced Wednesday that it sold a 32 percent stake in chemicals and fertilizer group Kemira Oyj for ( x20ac ) 655.6 million ( $ 890US million ) , sending the company 's share price up 6 percent . |
positive |
neutral (p = 0.75) |
| 230 |
Last year the company raised its turnover to approximately 7 million litas EUR 2 mln , from 6.1 million litas in 2004 . |
positive |
neutral (p = 0.47) |
| 235 |
Olvi , which controls a 94 percent stake in Ragutis through A. Le Coq , said in its annual report published earlier this year that the Lithuanian brewery 's sales reached 15.04 million euros last year , a rise of 20.4 percent from 12.49 million euros in 2004 . |
positive |
neutral (p = 0.58) |
For records in the dataset where text contains "said", the Precision is 11.06% lower than the global Precision.
| Level |
Data slice |
Metric |
Deviation |
| major 🔴 |
text contains "said" |
Precision = 0.654 |
-11.06% than global |
Taxonomy
avid-effect:performance:P0204
🔍✨Examples
|
text |
label |
Predicted label |
| 105 |
Digia said its consolidated net sales for January-June 2010 were EUR67 .8 m , up 9.7 % on the same period in 2009 ( EUR61 .9 m ) . |
positive |
neutral (p = 0.51) |
| 115 |
Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . |
positive |
neutral (p = 0.58) |
| 235 |
Olvi , which controls a 94 percent stake in Ragutis through A. Le Coq , said in its annual report published earlier this year that the Lithuanian brewery 's sales reached 15.04 million euros last year , a rise of 20.4 percent from 12.49 million euros in 2004 . |
positive |
neutral (p = 0.58) |
👉Ethical issues (2)
When feature “text” is perturbed with the transformation “Switch Gender”, the model changes its prediction in 6.15% of the cases. We expected the predictions not to be affected by this transformation.
| Level |
Metric |
Transformation |
Deviation |
| medium 🟡 |
Fail rate = 0.062 |
Switch Gender |
4/65 tested samples (6.15%) changed prediction after perturbation |
Taxonomy
avid-effect:ethics:E0101
avid-effect:performance:P0201
🔍✨Examples
|
text |
Switch Gender(text) |
Original prediction |
Prediction after perturbation |
| 246 |
Revenue grew 1 percent to euro742 .2 million US$ 964 million from euro735 million . |
revenu grew 1 percent to euro742 .2 million US$ 964 million from euro735 million . |
positive (p = 0.89) |
neutral (p = 0.46) |
| 1237 |
He joined Rautakirja in 1997 to lead one of its four business areas and took up the position of President and CEO in 2001 . |
she joined Rautakirja in 1997 to lead one of its four business areas and took up the position of President and CEO in 2001 . |
neutral (p = 0.58) |
positive (p = 0.50) |
| 1404 |
The company 's model is based on developer contracting construction projects and customer focused project management . |
The company 's mannequin is based on developer contracting construction projects and customer focused project management . |
positive (p = 0.58) |
neutral (p = 0.59) |
When feature “text” is perturbed with the transformation “Switch countries from high- to low-income and vice versa”, the model changes its prediction in 5.43% of the cases. We expected the predictions not to be affected by this transformation.
| Level |
Metric |
Transformation |
Deviation |
| medium 🟡 |
Fail rate = 0.054 |
Switch countries from high- to low-income and vice versa |
28/516 tested samples (5.43%) changed prediction after perturbation |
Taxonomy
avid-effect:ethics:E0101
avid-effect:performance:P0201
🔍✨Examples
|
text |
Switch countries from high- to low-income and vice versa(text) |
Original prediction |
Prediction after perturbation |
| 123 |
Finnish silicon wafer technology company Okmetic Oyj ( OMX Helsinki : OKM1V ) reported on Thursday ( 7 August ) an operating profit of EUR5 .3 m for the period January-June 2008 , up from EUR3 .3 m in the corresponding period in 2007 . |
Samoan silicon wafer technology company Okmetic Oyj ( OMX Helsinki : OKM1V ) reported on Thursday ( 7 August ) an operating profit of EUR5 .3 m for the period January-June 2008 , up from EUR3 .3 m in the corresponding period in 2007 . |
neutral (p = 0.47) |
positive (p = 0.47) |
| 148 |
( ADP News ) - Nov 5 , 2008 - Finnish electronic measurement products and solutions maker Vaisala Oyj ( OMX : VAIAS ) said today that its net profit rose to EUR 18 million ( USD 23.1 m ) for the first nine months of 2008 from EUR 1 |
( ADP News ) - Nov 5 , 2008 - Congolese electronic measurement products and solutions maker Vaisala Oyj ( OMX : VAIAS ) said today that its net profit rose to EUR 18 million ( USD 23.1 m ) for the first nine months of 2008 from EUR 1 |
positive (p = 0.52) |
neutral (p = 0.58) |
| 331 |
Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said today its net loss narrowed to EUR 500,000 ( USD 680,000 ) in the last quarter of 2010 from EUR 5.3 million for the same period a year earlier . |
Uzbekistani metal products company Componenta Oyj ( HEL : CTH1V ) said today its net loss narrowed to EUR 500,000 ( USD 680,000 ) in the last quarter of 2010 from EUR 5.3 million for the same period a year earlier . |
neutral (p = 0.43) |
negative (p = 0.42) |
Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.
Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.
