|
--- |
|
tags: |
|
- sentence-transformers |
|
- transformers |
|
- SetFit |
|
- News |
|
datasets: KnutJaegersberg/News_topics_IPTC_codes_long |
|
pipeline_tag: text-classification |
|
--- |
|
|
|
|
|
# IPTC topic classifier (multilingual) |
|
|
|
A SetFit model fit on 166 downlsampled multilingual IPTC Subject labels (concatenated for the lowest hierarchy level into artificial sentences of keywords) to predict the mid level news categories. |
|
The purpose of this classifier is to support exploring corpora as weak labeler, since the representations of these descriptions are only approximations of real documents from those topics. |
|
The dataset I used to train the model is based on this file: |
|
https://huggingface.co/datasets/KnutJaegersberg/News_topics_IPTC_codes_long |
|
|
|
Accuracy on highest level labels in eval: |
|
0.9779412 |
|
Accuracy/F1/mcc on mid level labels in eval: |
|
0.6992481/0.6666667/0.6992617 |
|
|
|
More interestingly, I used the kaggle dataset with headlines from huffington post and manually selected 15 overlapping high level categories to evaluate the performance. |
|
https://www.kaggle.com/datasets/rmisra/news-category-dataset |
|
|
|
While mcc 0.1968043 on this dataset does not sound as good as before, the mistakes usually could also be seen as a re-interpretation. I.e. news on arrests where categorized as entertainment in the huffington post dataset, the classifier put it into the crime category. |
|
My current impression is this system is useful for the aimed for purpose. |
|
|
|
|
|
|
|
The numeric categories can be joined with the labels by using this table: |
|
https://huggingface.co/datasets/KnutJaegersberg/IPTC-topic-classifier-labels |
|
|
|
|
|
Looks like try out api box to the right by huggingface does not yet handle setfit models, can't do anything about that. |
|
|
|
|
|
Use like any other SetFit model |
|
|
|
from setfit import SetFitModel |
|
|
|
# Download from Hub and run inference |
|
model = SetFitModel.from_pretrained("KnutJaegersberg/IPTC-classifier-ml") |
|
# Run inference |
|
preds = model(["Rachel Dolezal Faces Felony Charges For Welfare Fraud", "Elon Musk just got lucky", "The hype on AI is different from the hype on other tech topics"]) |
