File size: 13,717 Bytes
72e5bba b88cbb2 72e5bba bdeb9bb 72e5bba b88cbb2 72e5bba b88cbb2 72e5bba 3b2c9a6 12cb57d 68d7480 12cb57d bc43590 12cb57d 72e5bba 76517ea b88cbb2 72e5bba 12cb57d 72e5bba b88cbb2 84a23e1 4929dd1 e12e4f3 4929dd1 84a23e1 b88cbb2 72e5bba |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 |
---
license: mit
language:
- en
metrics:
- accuracy
- precision
- recall
model-index:
- name: PolicyBERTa-7d
results: []
widget:
- text: "Russia must end the war."
- text: "Democratic institutions must be supported."
- text: "The state must fight political corruption."
- text: "Our energy economy must be nationalised."
- text: "We must increase social spending."
---
# PolicyBERTa-7d
This model is a fine-tuned version of [roberta-base](https://huggingface.co/roberta-base) on data from the [Manifesto Project](https://manifesto-project.wzb.eu/). It was inspired by the model from [Laurer (2020)](https://huggingface.co/MoritzLaurer/policy-distilbert-7d).
It achieves the following results on the evaluation set:
- Loss: 0.8549
- Accuracy: 0.7059
- F1-micro: 0.7059
- F1-macro: 0.6683
- F1-weighted: 0.7033
- Precision: 0.7059
- Recall: 0.7059
## Model description
This model was trained on 115,943 manually annotated sentences to classify text into one of seven political categories: "external relations", "freedom and democracy", "political system", "economy", "welfare and quality of life", "fabric of society" and "social groups".
## Intended uses & limitations
The model output reproduces the limitations of the dataset in terms of country coverage, time span, domain definitions and potential biases of the annotators - as any supervised machine learning model would. Applying the model to other types of data (other types of texts, countries etc.) will reduce performance.
```python
from transformers import pipeline
import pandas as pd
classifier = pipeline(
task="text-classification",
model="niksmer/PolicyBERTa-7d")
# Load text data you want to classify
text = pd.read_csv("example.csv")["text_you_want_to_classify"].to_list()
# Inference
output = classifier(text)
# Print output
pd.DataFrame(output).head()
```
## Training and evaluation data
PolicyBERTa-7d was trained on the English-speaking subset of the [Manifesto Project Dataset (MPDS2021a)](https://manifesto-project.wzb.eu/datasets). The model was trained on 115,943 sentences from 163 political manifestos in 7 English-speaking countries (Australia, Canada, Ireland, New Zealand, South Africa, United Kingdom, United States). The manifestos were published between 1992 - 2020.
| Country | Count manifestos | Count sentences | Time span |
|----------------|------------------|-----------------|--------------------|
| Australia | 18 | 14,887 | 2010-2016 |
| Ireland | 23 | 24,966 | 2007-2016 |
| Canada | 14 | 12,344 | 2004-2008 & 2015 |
| New Zealand | 46 | 35,079 | 1993-2017 |
| South Africa | 29 | 13,334 | 1994-2019 |
| USA | 9 | 13,188 | 1992 & 2004-2020 |
| United Kingdom | 34 | 30,936 | 1997-2019 |
Canadian manifestos between 2004 and 2008 are used as test data.
The Manifesto Project mannually annotates individual sentences from political party manifestos in 7 main political domains: 'Economy', 'External Relations', 'Fabric of Society', 'Freedom and Democracy', 'Political System', 'Welfare and Quality of Life' or 'Social Groups' - see the [codebook](https://manifesto-project.wzb.eu/down/papers/handbook_2021_version_5.pdf) for the exact definitions of each domain.
### Tain data
Train data was higly imbalanced.
| Label | Description | Count |
|------------|--------------|--------|
| 0 | external relations | 7,640 |
| 1 | freedom and democracy | 5,880 |
| 2 | political system | 11,234 |
| 3 | economy | 29,218 |
| 4 | welfare and quality of life | 37,200 |
| 5 | fabric of society | 13,594 |
| 6 | social groups | 11,177 |
Overall count: 115,943
### Validation data
The validation was created by chance.
| Label | Description | Count |
|------------|--------------|--------|
| 0 | external relations | 1,345 |
| 1 | freedom and democracy | 1,043 |
| 2 | political system | 2,038 |
| 3 | economy | 5,140 |
| 4 | welfare and quality of life | 6,554 |
| 5 | fabric of society | 2,384 |
| 6 | social groups | 1,957 |
Overall count: 20,461
## Test data
The test dataset contains ten canadian manifestos between 2004 and 2008.
| Label | Description | Count |
|------------|--------------|--------|
| 0 | external relations | 824 |
| 1 | freedom and democracy | 296 |
| 2 | political system | 1,041 |
| 3 | economy | 2,188 |
| 4 | welfare and quality of life | 2,654 |
| 5 | fabric of society | 940 |
| 6 | social groups | 387 |
Overall count: 8,330
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
```
training_args = TrainingArguments(
warmup_steps=0,
weight_decay=0.1,
learning_rate=1e-05,
fp16 = True,
evaluation_strategy="epoch",
num_train_epochs=5,
per_device_train_batch_size=16,
overwrite_output_dir=True,
per_device_eval_batch_size=16,
save_strategy="no",
logging_dir='logs',
logging_strategy= 'steps',
logging_steps=10,
push_to_hub=True,
hub_strategy="end")
```
### Training results
| Training Loss | Epoch | Step | Validation Loss | Accuracy | F1-micro | F1-macro | F1-weighted | Precision | Recall |
|:-------------:|:-----:|:----:|:---------------:|:--------:|:--------:|:--------:|:-----------:|:---------:|:------:|
| 0.9154 | 1.0 | 1812 | 0.8984 | 0.6785 | 0.6785 | 0.6383 | 0.6772 | 0.6785 | 0.6785 |
| 0.8374 | 2.0 | 3624 | 0.8569 | 0.6957 | 0.6957 | 0.6529 | 0.6914 | 0.6957 | 0.6957 |
| 0.7053 | 3.0 | 5436 | 0.8582 | 0.7019 | 0.7019 | 0.6594 | 0.6967 | 0.7019 | 0.7019 |
| 0.7178 | 4.0 | 7248 | 0.8488 | 0.7030 | 0.7030 | 0.6662 | 0.7011 | 0.7030 | 0.7030 |
| 0.6688 | 5.0 | 9060 | 0.8549 | 0.7059 | 0.7059 | 0.6683 | 0.7033 | 0.7059 | 0.7059 |
### Validation evaluation
| Model | Micro F1-Score | Macro F1-Score | Weighted F1-Score |
|----------------|----------------|----------------|-------------------|
| PolicyBERTa-7d | 0.71 | 0.67 | 0.70 |
### Test evaluation
| Model | Micro F1-Score | Macro F1-Score | Weighted F1-Score |
|----------------|----------------|----------------|-------------------|
| PolicyBERTa-7d | 0.65 | 0.60 | 0.65 |
### Evaluation per category
| Label | Validation F1-Score | Test F1-Score |
|-----------------------------|---------------------|---------------|
| external relations | 0.76 | 0.70 |
| freedom and democracy | 0.61 | 0.55 |
| political system | 0.55 | 0.55 |
| economy | 0.74 | 0.67 |
| welfare and quality of life | 0.77 | 0.72 |
| fabric of society | 0.67 | 0.60 |
| social groups | 0.58 | 0.41 |
### Evaluation based on saliency theory
Saliency theory is a theory to analyse politial text data. In sum, parties tend to write about policies in which they think that they are seen as competent.
Voters tend to assign advantages in policy competence in line to the assumed ideology of parties. Therefore you can analyze the share of policies parties tend to write about in their manifestos to analyze the party ideology.
The Manifesto Project presented for such an analysis the rile-index. For a quick overview, check [this](https://manifesto-project.wzb.eu/down/tutorials/main-dataset.html#measuring-parties-left-right-positions). But PolicyBERTa isn't fine-tuned to predict the rile-index, if you're interested in that, check [ManiBERT](https://huggingface.co/niksmer/ManiBERT) or [RoBERTa-RILE](https://huggingface.co/niksmer/RoBERTa-RILE).
In the following table, the predicted and original share of the individual policy domains are shown per manifesto in the test dataset. Overall the pearson correlation between the predicted and original shares is 0.965.
| Party-ID | Year | Type | Share external relations | Share freedom and democracy | Share political system | Share economy | Share welfare and quality of life | Share fabric of society | Share social groups |
|--------------|-------------|---------------|--------------------------|-----------------------------|------------------------|----------------|-----------------------------------|-------------------------|---------------------|
| 62320 | 2004 | Predicted | 7.1% | 4.8% | 13.2% | 20.3% | 35.2% | 9.6% | 9.8% |
| | | Original | 10.2% | 2.5% | 13.7% | 23.8% | 31.7% | 11.6% | 6.4% |
| 62320 | 2006 | Predicted | 2.9% | 4.7% | 16.4% | 18.9% | 38.3% | 11.9% | 6.9% |
| | | Original | 5.6% | 5.0% | 15.8% | 20.7% | 38.7% | 9.3% | 4.9% |
| 62320 | 2008 | Predicted | 6.8% | 4.7% | 6.2% | 24.7% | 38.3% | 10.3% | 9.0% |
| | | Original | 5.6% | 3.7% | 8.2% | 33.1% | 29.5% | 11.7% | 4.3% |
| 62420 | 2004 | Predicted | 9.7% | 3.5% | 14.5% | 24.7% | 34.8% | 8.5% | 4.3% |
| | | Original | 12.6% | 1.3% | 18.8% | 23.0% | 33.2% | 9.0% | 2.0% |
| 62420 | 2006 | Predicted | 9.5% | 2.2% | 7.9% | 27.8% | 34.8% | 9.2% | 8.7% |
| | | Original | 10.6% | 2.5% | 9.6% | 29.7% | 33.1% | 8.3% | 6.2% |
| 62420 | 2008 | Predicted | 0.7% | 0.5% | 3.5% | 41.7% | 46.4% | 3.7% | 3.5% |
| | | Original | 2.0% | 0.2% | 4.4% | 33.3% | 45.9% | 7.7% | 6.4% |
| 62623 | 2004 | Predicted | 7.1% | 11.4% | 24.5% | 17.6% | 21.5% | 13.6% | 4.3% |
| | | Original | 8.4% | 6.7% | 28.8% | 17.4% | 18.7% | 15.5% | 4.5% |
| 62623 | 2006 | Predicted | 5.6% | 8.5% | 23.6% | 15.6% | 14.8% | 24.3% | 7.6% |
| | | Original | 5.0% | 8.9% | 22.2% | 17.4% | 17.2% | 25.7% | 3.6% |
| 62623 | 2008 | Predicted | 5.0% | 4.4% | 12.2% | 33.1% | 21.9% | 17.5% | 5.9% |
| | | Original | 5.6% | 2.2% | 11.6% | 37.8% | 17.8% | 20.9% | 4.1% |
| 62110 | 2008 | Predicted | 10.0% | 3.1% | 6.8% | 22.7% | 41.3% | 10.1% | 6.0% |
| | | Original | 13.4% | 3.3% | 7.7% | 26.9% | 35.6% | 8.9% | 4.3% |
### Framework versions
- Transformers 4.16.2
- Pytorch 1.9.0+cu102
- Datasets 1.8.0
- Tokenizers 0.10.3
|