|
--- |
|
license: mit |
|
language: |
|
- en |
|
metrics: |
|
- accuracy |
|
- precision |
|
- recall |
|
model-index: |
|
- name: PolicyBERTa-7d |
|
results: [] |
|
widget: |
|
- text: "Russia must end the war." |
|
- text: "Democratic institutions must be supported." |
|
- text: "The state must fight political corruption." |
|
- text: "Our energy economy must be nationalised." |
|
- text: "We must increase social spending." |
|
|
|
--- |
|
|
|
# PolicyBERTa-7d |
|
|
|
This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on data from the [Manifesto Project](https://manifesto-project.wzb.eu/). It was inspired by the model from [Laurer (2020)](https://huggingface.co/MoritzLaurer/policy-distilbert-7d). |
|
|
|
It achieves the following results on the evaluation set: |
|
- Loss: 0.8549 |
|
- Accuracy: 0.7059 |
|
- F1-micro: 0.7059 |
|
- F1-macro: 0.6683 |
|
- F1-weighted: 0.7033 |
|
- Precision: 0.7059 |
|
- Recall: 0.7059 |
|
|
|
## Model description |
|
|
|
This model was trained on 115,943 manually annotated sentences to classify text into one of seven political categories: "external relations", "freedom and democracy", "political system", "economy", "welfare and quality of life", "fabric of society" and "social groups". |
|
|
|
|
|
## Intended uses & limitations |
|
|
|
The model output reproduces the limitations of the dataset in terms of country coverage, time span, domain definitions and potential biases of the annotators - as any supervised machine learning model would. Applying the model to other types of data (other types of texts, countries etc.) will reduce performance. |
|
|
|
```python |
|
from transformers import pipeline |
|
import pandas as pd |
|
|
|
classifier = pipeline( |
|
task="text-classification", |
|
model="niksmer/PolicyBERTa-7d") |
|
|
|
# Load text data you want to classify |
|
text = pd.read_csv("text.csv") |
|
|
|
# Inference |
|
output = classifier(text) |
|
|
|
# Print output |
|
pd.DataFrame(output).head() |
|
``` |
|
|
|
## Training and evaluation data |
|
|
|
PolicyBERTa-7d was trained on the English-speaking subset of the [Manifesto Project Dataset (MPDS2020a)](https://manifesto-project.wzb.eu/datasets). The model was trained on 115,943 sentences from 163 political manifestos in 7 English-speaking countries (Australia, Canada, Ireland, New Zealand, South Africa, United Kingdom, United States). The manifestos were published between 1992 - 2020. |
|
|
|
| Country | Count manifestos | Count sentences | Time span | |
|
|----------------|------------------|-----------------|--------------------| |
|
| Australia | 18 | 14,887 | 2010-2016 | |
|
| Ireland | 23 | 24,966 | 2007-2016 | |
|
| Canada | 14 | 12,344 | 2004-2008 & 2015 | |
|
| New Zealand | 46 | 35,079 | 1993-2017 | |
|
| South Africa | 29 | 13,334 | 1994-2019 | |
|
| USA | 9 | 13,188 | 1992 & 2004-2020 | |
|
| United Kingdom | 34 | 30,936 | 1997-2019 | |
|
|
|
Canadian manifestos between 2004 and 2008 are used as test data. |
|
|
|
|
|
The Manifesto Project mannually annotates individual sentences from political party manifestos in 7 main political domains: 'Economy', 'External Relations', 'Fabric of Society', 'Freedom and Democracy', 'Political System', 'Welfare and Quality of Life' or 'Social Groups' - see the [codebook](https://manifesto-project.wzb.eu/down/papers/handbook_2021_version_5.pdf) for the exact definitions of each domain. |
|
|
|
### Tain data |
|
|
|
Train data was higly imbalanced. |
|
|
|
| Label | Description | Count | |
|
|------------|--------------|--------| |
|
| 0 | external relations | 7,640 | |
|
| 1 | freedom and democracy | 5,880 | |
|
| 2 | political system | 11,234 | |
|
| 3 | economy | 29,218 | |
|
| 4 | welfare and quality of life | 37,200 | |
|
| 5 | fabric of society | 13,594 | |
|
| 6 | social groups | 11,177 | |
|
|
|
Overall count: 115,943 |
|
|
|
### Validation data |
|
|
|
The validation was created by chance. |
|
|
|
| Label | Description | Count | |
|
|------------|--------------|--------| |
|
| 0 | external relations | 1,345 | |
|
| 1 | freedom and democracy | 1,043 | |
|
| 2 | political system | 2,038 | |
|
| 3 | economy | 5,140 | |
|
| 4 | welfare and quality of life | 6,554 | |
|
| 5 | fabric of society | 2,384 | |
|
| 6 | social groups | 1,957 | |
|
|
|
Overall count: 20,461 |
|
|
|
## Test data |
|
|
|
The test dataset contains ten canadian manifestos between 2004 and 2008. |
|
|
|
| Label | Description | Count | |
|
|------------|--------------|--------| |
|
| 0 | external relations | 824 | |
|
| 1 | freedom and democracy | 296 | |
|
| 2 | political system | 1,041 | |
|
| 3 | economy | 2,188 | |
|
| 4 | welfare and quality of life | 2,654 | |
|
| 5 | fabric of society | 940 | |
|
| 6 | social groups | 387 | |
|
|
|
Overall count: 8,330 |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
``` |
|
training_args = TrainingArguments( |
|
warmup_steps=0, |
|
weight_decay=0.1, |
|
learning_rate=1e-05, |
|
fp16 = True, |
|
evaluation_strategy="epoch", |
|
num_train_epochs=5, |
|
per_device_train_batch_size=16, |
|
overwrite_output_dir=True, |
|
per_device_eval_batch_size=16, |
|
save_strategy="no", |
|
logging_dir='logs', |
|
logging_strategy= 'steps', |
|
logging_steps=10, |
|
push_to_hub=True, |
|
hub_strategy="end") |
|
``` |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1-micro | F1-macro | F1-weighted | Precision | Recall | |
|
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:--------:|:-----------:|:---------:|:------:| |
|
| 0.9154 | 1.0 | 1812 | 0.8984 | 0.6785 | 0.6785 | 0.6383 | 0.6772 | 0.6785 | 0.6785 | |
|
| 0.8374 | 2.0 | 3624 | 0.8569 | 0.6957 | 0.6957 | 0.6529 | 0.6914 | 0.6957 | 0.6957 | |
|
| 0.7053 | 3.0 | 5436 | 0.8582 | 0.7019 | 0.7019 | 0.6594 | 0.6967 | 0.7019 | 0.7019 | |
|
| 0.7178 | 4.0 | 7248 | 0.8488 | 0.7030 | 0.7030 | 0.6662 | 0.7011 | 0.7030 | 0.7030 | |
|
| 0.6688 | 5.0 | 9060 | 0.8549 | 0.7059 | 0.7059 | 0.6683 | 0.7033 | 0.7059 | 0.7059 | |
|
|
|
### Validation evaluation |
|
|
|
| Model | Micro F1-Score | Macro F1-Score | Weighted F1-Score | |
|
|----------------|----------------|----------------|-------------------| |
|
| PolicyBERTa-7d | 0.71 | 0.67 | 0.70 | |
|
|
|
|
|
|
|
### Test evaluation |
|
|
|
| Model | Micro F1-Score | Macro F1-Score | Weighted F1-Score | |
|
|----------------|----------------|----------------|-------------------| |
|
| PolicyBERTa-7d | 0.65 | 0.60 | 0.65 | |
|
|
|
|
|
### Evaluation per category |
|
|
|
| Label | Validation F1-Score | Test F1-Score | |
|
|-----------------------------|---------------------|---------------| |
|
| external relations | 0.76 | 0.70 | |
|
| freedom and democracy | 0.61 | 0.55 | |
|
| political system | 0.55 | 0.55 | |
|
| economy | 0.74 | 0.67 | |
|
| welfare and quality of life | 0.77 | 0.72 | |
|
| fabric of society | 0.67 | 0.60 | |
|
| social groups | 0.58 | 0.41 | |
|
|
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.16.2 |
|
- Pytorch 1.9.0+cu102 |
|
- Datasets 1.8.0 |
|
- Tokenizers 0.10.3 |
|
|
