cardiffnlp/twitter-roberta-base-sentiment-latest · Report for cardiffnlp/twitter-roberta-base-sentiment-latest

Hi Team,

This is a report from Giskard Bot Scan 🐢.

We have identified 8 potential vulnerabilities in your model based on an automated scan.

This automated analysis evaluated the model on the dataset tweet_eval (subset sentiment, split train).

You can find a full version of scan report here.

👉Performance issues (1)

For records in the dataset where text contains "like", the Precision is 5.91% lower than the global Precision.

Level	Data slice	Metric	Deviation
medium 🟡	`text` contains "like"	Precision = 0.731	-5.91% than global

Taxonomy

avid-effect:performance:P0204

🔍✨Examples

	text	label	Predicted `label`
19	What is Jamie Foxx doing sitting next to Niall like you could've sat in a better spot just saying lol	neutral	negative (p = 0.49)
33	Right guys\u002c last competition of the night... Like this status for a chance to win a copy of Judas Priest\u2019s 30th...	positive	neutral (p = 0.74)
202	"Where's the sun mommy?" "It's asleep like you should be" "But Harper's a moon mommy" Dammit kid.	neutral	negative (p = 0.47)

👉Robustness issues (5)

When feature “text” is perturbed with the transformation “Transform to uppercase”, the model changes its prediction in 16.5% of the cases. We expected the predictions not to be affected by this transformation.

Level	Metric	Transformation	Deviation
major 🔴	Fail rate = 0.165	Transform to uppercase	165/1000 tested samples (16.5%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201

🔍✨Examples

	text	Transform to uppercase(text)	Original prediction	Prediction after perturbation
12387	Starting work on Fashion Star tomorrow! My couch is really going to miss me. I got myself a treat to celebrate:	STARTING WORK ON FASHION STAR TOMORROW! MY COUCH IS REALLY GOING TO MISS ME. I GOT MYSELF A TREAT TO CELEBRATE:	positive (p = 0.90)	negative (p = 0.43)
15177	"my English teacher has very long, thick hair and I hope he puts it up in leias buns for Star Wars day at school tomorrow"	"MY ENGLISH TEACHER HAS VERY LONG, THICK HAIR AND I HOPE HE PUTS IT UP IN LEIAS BUNS FOR STAR WARS DAY AT SCHOOL TOMORROW"	neutral (p = 0.48)	positive (p = 0.66)
16163	Saudi Arabia's oil-induced cash crunch may come quicker than you think	SAUDI ARABIA'S OIL-INDUCED CASH CRUNCH MAY COME QUICKER THAN YOU THINK	negative (p = 0.57)	neutral (p = 0.67)

When feature “text” is perturbed with the transformation “Add typos”, the model changes its prediction in 12.5% of the cases. We expected the predictions not to be affected by this transformation.

Level	Metric	Transformation	Deviation
major 🔴	Fail rate = 0.125	Add typos	125/1000 tested samples (12.5%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201

🔍✨Examples

	text	Add typos(text)	Original prediction	Prediction after perturbation
20913	"Don't think there will be a new episode of Naruto tomorrow. Welp, I'll just have to wait till Friday to watch TV."	"Doh't think there will be a new episode of Naruto toomrrow. Welp, I'll just have to wait gill Friday to watch TV."	neutral (p = 0.49)	negative (p = 0.55)
19823	Ed Sheeran may make lovely music but he looks like a 'sex-case'	Ed Sheeran ma make lovely music byt e looks like a 'sexcase'	negative (p = 0.63)	positive (p = 0.87)
35893	Katie Hopkins turns her nose up at people but.. 1. Writes for the Sun 2. Was on Celebrity Big Brother 3. This programme #KatieRules	Jatie Hopkis turns her nose up at people ubt.. 1. Writex for the Sun 2. Was on Celebrity Big Brothe 3. This programjde #KatieRules	neutral (p = 0.66)	negative (p = 0.56)

When feature “text” is perturbed with the transformation “Transform to title case”, the model changes its prediction in 11.0% of the cases. We expected the predictions not to be affected by this transformation.

Level	Metric	Transformation	Deviation
major 🔴	Fail rate = 0.110	Transform to title case	110/1000 tested samples (11.0%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201

🔍✨Examples

	text	Transform to title case(text)	Original prediction	Prediction after perturbation
29181	"Kris Bryant is on fire homering twice, finishing with three RBI and four runs scored in Sunday's 9-3 win over the Braves. #Cubs #MLB"	"Kris Bryant Is On Fire Homering Twice, Finishing With Three Rbi And Four Runs Scored In Sunday'S 9-3 Win Over The Braves. #Cubs #Mlb"	positive (p = 0.62)	neutral (p = 0.54)
16163	Saudi Arabia's oil-induced cash crunch may come quicker than you think	Saudi Arabia'S Oil-Induced Cash Crunch May Come Quicker Than You Think	negative (p = 0.57)	neutral (p = 0.78)
43340	Sarah G. Day last Sunday!! Monday!! Tuesday!! Wednesday!! & xempre ngaung Thursday!! kip it up POPTSERS!!<3	Sarah G. Day Last Sunday!! Monday!! Tuesday!! Wednesday!! & Xempre Ngaung Thursday!! Kip It Up Poptsers!!<3	positive (p = 0.63)	neutral (p = 0.54)

When feature “text” is perturbed with the transformation “Punctuation Removal”, the model changes its prediction in 8.0% of the cases. We expected the predictions not to be affected by this transformation.

Level	Metric	Transformation	Deviation
medium 🟡	Fail rate = 0.080	Punctuation Removal	80/1000 tested samples (8.0%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201

🔍✨Examples

	text	Punctuation Removal(text)	Original prediction	Prediction after perturbation
27468	@user <sigh> Yes I'll still be painting your Batman minis. Although I may need to take a break to make sure I get my Malifaux done :)	@user &lt sigh&gt Yes I ll still be painting your Batman minis Although I may need to take a break to make sure I get my Malifaux done )	positive (p = 0.59)	neutral (p = 0.50)
20830	@user The bookstore is packed with students buying warm WVU gear for the game this Saturday! #215section1	@user The bookstore is packed with students buying warm WVU gear for the game this Saturday #215section1	positive (p = 0.67)	neutral (p = 0.60)
32555	The what's a better show debate, Seinfeld vs Friends is one for the ages but you may be mentally incompetent if you think Friends is better.	The what s a better show debate Seinfeld vs Friends is one for the ages but you may be mentally incompetent if you think Friends is better	neutral (p = 0.42)	negative (p = 0.43)

When feature “text” is perturbed with the transformation “Transform to lowercase”, the model changes its prediction in 5.6% of the cases. We expected the predictions not to be affected by this transformation.

Level	Metric	Transformation	Deviation
medium 🟡	Fail rate = 0.056	Transform to lowercase	56/1000 tested samples (5.6%) changed prediction after perturbation

Taxonomy

avid-effect:performance:P0201

🔍✨Examples

	text	Transform to lowercase(text)	Original prediction	Prediction after perturbation
23559	"Got fiends throwin up on themselves like Willie Beamon. Any Given Sunday, Gunplays Optional. However a nigga want it like Soul La Soul."	"got fiends throwin up on themselves like willie beamon. any given sunday, gunplays optional. however a nigga want it like soul la soul."	neutral (p = 0.64)	negative (p = 0.58)
44141	@user Have you heard any reasoned justification for making James Pattinson 12th man?	@user have you heard any reasoned justification for making james pattinson 12th man?	neutral (p = 0.54)	negative (p = 0.52)
18195	@user @user May the merciful Lord give them victory over d Boko Haram the murderers	@user @user may the merciful lord give them victory over d boko haram the murderers	neutral (p = 0.49)	negative (p = 0.49)

👉Underconfidence issues (1)

For records in your dataset where text contains "like", we found a significantly higher number of underconfident predictions (68 samples, corresponding to 2.86% of the predictions in the data slice).

Level	Data slice	Metric	Deviation
major 🔴	`text` contains "like"	Underconfidence rate = 0.029	+56.22% than global

Taxonomy

avid-effect:performance:P0204

🔍✨Examples

	text	label	Predicted `label`
33678	@user @user @user @user whys he flossin an iPhone 3G, original iPad an 5th gen MacBook like he got money lmao	negative	neutral (p = 0.47)
			negative (p = 0.47)
27874	"#nw kung fu panda 2. dnt judge me, i like animated movies (plus the 1st one was the shit)."	positive	neutral (p = 0.36)
			positive (p = 0.36)
40419	codigames:Do you like ARPGs?Dungeon Legends arrive to iOS on September 3rd #gamedev #iphone	positive	positive (p = 0.50)
			neutral (p = 0.50)

👉Ethical issues (1)

When feature “text” is perturbed with the transformation “Switch Gender”, the model changes its prediction in 5.2% of the cases. We expected the predictions not to be affected by this transformation.

Level	Metric	Transformation	Deviation
medium 🟡	Fail rate = 0.052	Switch Gender	52/1000 tested samples (5.2%) changed prediction after perturbation

Taxonomy

avid-effect:ethics:E0101 avid-effect:performance:P0201

🔍✨Examples

	text	Switch Gender(text)	Original prediction	Prediction after perturbation
29000	# An 80 years old Bentley belongs to my Brother and Sister in Law Only one in the world 4th Owner...	# An 80 years old Bentley belongs to my sister and brother in Law Only one in the world 4th Owner...	positive (p = 0.50)	neutral (p = 0.57)
9082	Richie Hawtin - Monday Night - Buffalo Town Ballroom - Don\u2019t miss it!	Richie Hawtin - Monday Night - Buffalo Town Ballroom - Don\u2019t mr. it!	positive (p = 0.92)	neutral (p = 0.80)
3199	"Admit it. You may hate John Cena, but you know you miss his U.S. open challenges!"	"Admit it. You may hate John Cena, but you know you mr. her U.S. open challenges!"	negative (p = 0.46)	neutral (p = 0.50)

Checkout out the Giskard Space and Giskard Documentation to learn more about how to test your model.

Disclaimer: it's important to note that automated scans may produce false positives or miss certain vulnerabilities. We encourage you to review the findings and assess the impact accordingly.