Report for yiyanghkust/finbert-tone

#193
by giskard-bot - opened
Giskard org

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 8 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset financial_phrasebank (subset sentences_allagree, split train).

You can find a full version of scan report here.

👉Performance issues (7)

For records in the dataset where text contains "said", the Precision is 19.67% lower than the global Precision.

Level Data slice Metric Deviation
major 🔴 text contains "said" Precision = 0.737 -19.67% than global

Taxonomy

avid-effect:performance:P0204
🔍✨Examples
text label Predicted label
115 Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . Positive Neutral (p = 1.00)
129 However , sales returned to growth in April-June 2010 , CEO Pekka Eloholma said . Positive Neutral (p = 0.46)
312 Finnish telecoms software developer Tecnomen Oyj said on December 5 , 2006 it won a $ 3.3 mln ( 2.5 mln euro ) order to supply an expansion of the charging and messaging systems for the mobile and fixed networks of Brasil Telecom . Positive Neutral (p = 0.98)

For records in the dataset where text contains "period", the Precision is 10.49% lower than the global Precision.

Level Data slice Metric Deviation
major 🔴 text contains "period" Precision = 0.821 -10.49% than global

Taxonomy

avid-effect:performance:P0204
🔍✨Examples
text label Predicted label
47 Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . Positive Neutral (p = 0.74)
329 26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier . Positive Negative (p = 0.59)
330 8 May 2009 - Finnish liquid handling products and diagnostic test systems maker Biohit Oyj ( HEL : BIOBV ) said today ( 8 May 2009 ) its net loss narrowed to EUR0 .1 m ( USD0 .14 m ) for the first quarter of 2009 from EUR0 .4 m for the same period of 2008 . Positive Negative (p = 0.63)

For records in the dataset where text contains "finland", the Balanced Accuracy is 8.58% lower than the global Balanced Accuracy.

Level Data slice Metric Deviation
medium 🟡 text contains "finland" Balanced Accuracy = 0.810 -8.58% than global

Taxonomy

avid-effect:performance:P0204
🔍✨Examples
text label Predicted label
115 Finland-based Elcoteq SE , a privately held provider of electronics manufacturing services to communications companies , said Thursday it signed a long-term manufacturing supply deal with communications equipment company Andrew Corp. . Positive Neutral (p = 1.00)
324 In addition , Cramo and Peab have signed exclusive five-year rental agreements in Finland and have extended their existing rental agreements in the Swedish market for another five years . Positive Neutral (p = 1.00)
425 Citigroup , Inc ( NYSE : C ) has announced that its Global Transaction Services ( GTS ) business has been awarded a new mandate by Finland-based Pohjola Bank Group . Positive Neutral (p = 0.75)

For records in the dataset where text contains "year", the Precision is 6.02% lower than the global Precision.

Level Data slice Metric Deviation
medium 🟡 text contains "year" Precision = 0.862 -6.02% than global

Taxonomy

avid-effect:performance:P0204
🔍✨Examples
text label Predicted label
230 Last year the company raised its turnover to approximately 7 million litas EUR 2 mln , from 6.1 million litas in 2004 . Positive Neutral (p = 0.63)
321 LONDON MarketWatch -- Nokia nok said it 's won a five-year services contract to run Hutchison Essar 's network operations in nine locations in India . Positive Neutral (p = 1.00)
324 In addition , Cramo and Peab have signed exclusive five-year rental agreements in Finland and have extended their existing rental agreements in the Swedish market for another five years . Positive Neutral (p = 1.00)

For records in the dataset where text contains "eur", the Precision is 5.68% lower than the global Precision.

Level Data slice Metric Deviation
medium 🟡 text contains "eur" Precision = 0.865 -5.68% than global

Taxonomy

avid-effect:performance:P0204
🔍✨Examples
text label Predicted label
21 The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million . Positive Neutral (p = 1.00)
47 Finnish financial group Aktia reports operating profit of EUR 44.4 mn in January-September 2009 , up from EUR 37.3 mn in the corresponding period in 2008 . Positive Neutral (p = 0.74)
200 The fair value of the investment properties totaled EUR 2,534.9 mn , up from EUR 2,455.1 mn in 2006 . Positive Neutral (p = 1.00)

For records in the dataset where text contains "quarter", the Precision is 5.57% lower than the global Precision.

Level Data slice Metric Deviation
medium 🟡 text contains "quarter" Precision = 0.866 -5.57% than global

Taxonomy

avid-effect:performance:P0204
🔍✨Examples
text label Predicted label
328 In the second quarter of 2010 , the group 's pretax loss narrowed to EUR 400,000 from EUR 600,000 . Positive Negative (p = 0.98)
329 26 January 2011 - Finnish metal products company Componenta Oyj ( HEL : CTH1V ) said yesterday its net loss narrowed to EUR500 ,000 in the last quarter of 2010 from EUR5 .3 m for the same period a year earlier . Positive Negative (p = 0.59)
330 8 May 2009 - Finnish liquid handling products and diagnostic test systems maker Biohit Oyj ( HEL : BIOBV ) said today ( 8 May 2009 ) its net loss narrowed to EUR0 .1 m ( USD0 .14 m ) for the first quarter of 2009 from EUR0 .4 m for the same period of 2008 . Positive Negative (p = 0.63)

For records in the dataset where text_length(text) >= 122.500 AND text_length(text) < 133.500, the Balanced Accuracy is 5.28% lower than the global Balanced Accuracy.

Level Data slice Metric Deviation
medium 🟡 text_length(text) >= 122.500 AND text_length(text) < 133.500 Balanced Accuracy = 0.839 -5.28% than global

Taxonomy

avid-effect:performance:P0204
🔍✨Examples
text text_length(text) label Predicted label
10 Its board of directors will propose a dividend of EUR0 .12 per share for 2010 , up from the EUR0 .08 per share paid in 2009 . 125 Positive Neutral (p = 1.00)
21 The fair value of the property portfolio doubled as a result of the Kapiteeli acquisition and totalled EUR 2,686.2 1,259.7 million . 132 Positive Neutral (p = 1.00)
319 Bilfinger investors cheered the agreement , pushing shares up 7 % , or & euro ; 3.30 , to & euro ; 50.29 , in afternoon trade . 127 Positive Neutral (p = 1.00)
👉Robustness issues (1)

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 9.1% of the cases. We expected the predictions not to be affected by this transformation.

Level Metric Transformation Deviation
medium 🟡 Fail rate = 0.091 Add typos 91/1000 tested samples (9.1%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201
🔍✨Examples
text Add typos(text) Original prediction Prediction after perturbation
1143 A huge issue for us is the button placement . A huge isue for us is the button placment . Negative (p = 1.00) Neutral (p = 0.99)
2217 Talentum 's net sales in September were smaller than expected . Talentum ' net sales in September were skaler tham expeted . Negative (p = 1.00) Neutral (p = 1.00)
956 Elcoteq has a global network of After Market Service sites which have a long experience in serving Consumer Electronics and Systems Solutions customers . Elcoteq has a globa network of After Market Service sites which have a long experience in serving Consumer Electrinixs and Systems Solutions ustomers . Positive (p = 0.55) Neutral (p = 0.97)

Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.

Sign up or log in to comment