README.md

---
language: en
---
# UnifiedQA-Reddit-SYAC  

This is an abstractive title answering (TA) / clickbait spoiling model.  
This is a variant of [allenai/unifiedqa-t5-large](https://huggingface.co/allenai/unifiedqa-t5-large), fine-tuned on the Reddit SYAC dataset.  
The model was trained as part of my masters thesis:

_Abstractive title answering for clickbait content_  


### Disinformation  
This model has the proven capability of generating, and hallucinating false information.  
Any use of a TA system such as this one should be with knowledge of this risk.


## Performance  

### Intrinsic  

The following scores is the result of intrinsic evaluation on the Reddit SYAC test set.  
We used a max input length of 2048 and truncated the tokens exceeding this limit.  

| rouge1    | rouge2    | rougeL    | bleu      | meteor   |
|:----------|:----------|:----------|:----------|:---------|
| **44.58** | **23.89** | **43.45** | 17.46     | 36.22    |


### Qualtiy  
Using human evaluation, we measured model performance by asking the evaluators to rate the models
on a scale from 1 to 5 on how good their generated answer was for a given clickbait article.  

Mean quality = 4.065  

### Factuality  
We included a factuality assessment to address the issue of generating false information.  
Human raters were asked to place each output in the categories "True", "Irrelevant", and "False".  

| True    | Irrelevant | False    |
|:-------:|:----------:|:--------:|
|   85%   |    7.5%    | 7.5%     |

## Cite  

If you use this model, please cite my master's thesis

```
@mastersthesis{heiervang2022AbstractiveTA
  title={Abstractive title answering for clickbait content},
  author={Markus Sverdvik Heiervang},
  publisher={University of Oslo, Department of Informatics},
  year={2022}
}
```