|
--- |
|
license: eupl-1.1 |
|
datasets: |
|
- EuropeanParliament/cellar_eurovoc |
|
language: |
|
- en |
|
metrics: |
|
- type: f1 |
|
value: 0.8188 |
|
name: micro F1 |
|
args: |
|
threshold: 0.41 |
|
- type: NDCG@3 |
|
value: 0.8778 |
|
name: NDCG@5 |
|
- type: NDCG@5 |
|
value: 0.8625 |
|
name: NDCG@5 |
|
- type: NDCG@10 |
|
value: 0.8711 |
|
name: NDCG@10 |
|
tags: |
|
- eurovoc |
|
pipeline_tag: text-classification |
|
|
|
widget: |
|
- text: "The Union condemns the continuing grave human rights violations by the Myanmar armed forces, including torture, sexual and gender-based violence, the persecution of civil society actors, human rights defenders and journalists, and attacks on the civilian population, including ethnic and religious minorities." |
|
|
|
--- |
|
|
|
# Eurovoc Multilabel Classifer 🇪🇺 |
|
|
|
[EuroVoc](https://op.europa.eu/fr/web/eu-vocabularies) is a large multidisciplinary multilingual (24 languages of 🇪🇺) hierarchical thesaurus of more than 7000 classes covering the activities of EU institutions. |
|
Given the number of legal documents produced every day and the huge mass of pre-existing documents to be classified high quality automated or semi-automated classification methods are most welcome in this domain. |
|
|
|
This model based on BERT Deep Neural Network was trained on more than 3, 200,000 documents to achieve that task and is used in a production environment via the huggingface inference endpoint. |
|
This model support the 24 languages of the European Union. |
|
|
|
|
|
## Examples |
|
|
|
In English 🇬🇧 : |
|
|
|
``` |
|
text = "The Union condemns the continuing grave human rights violations by the Myanmar armed forces, including torture, sexual and gender-based violence, the persecution of civil society actors, human rights defenders and journalists, and attacks on the civilian population, including ethnic and religious minorities." |
|
|
|
human rights 0.9992846846580505 |
|
religious discrimination 0.8896209001541138 |
|
Burma/Myanmar 0.8654656410217285 |
|
EU relations 0.48358115553855896 |
|
sexual discrimination 0.3569115102291107 |
|
Christianity 0.2744523584842682 |
|
freedom of religious beliefs 0.21020834147930145 |
|
protection of minorities 0.11292753368616104 |
|
``` |
|
|
|
In French 🇫🇷: |
|
|
|
``` |
|
text = "En juillet 2023, la Commission a présenté un paquet de propositions pour l'écologisation du transport de marchandises. Parmi les trois propositions, l'une porte sur l'amélioration de l'utilisation des capacités de l'infrastructure ferroviaire. Le texte proposé comprend des modifications des règles relatives à la planification et à la répartition des capacités d'infrastructure ferroviaire, actuellement couvertes par la directive 2012/34/UE et le règlement (UE) n° 913/2010. L'objectif de ces modifications est de permettre une gestion plus efficace des capacités de l'infrastructure ferroviaire et du trafic, afin d'améliorer la qualité des services et d'optimiser l'utilisation du réseau ferroviaire, d'accueillir des volumes de trafic plus importants et de veiller à ce que le secteur des transports contribue à la décarbonisation." |
|
|
|
transport infrastructure 0.998161256313324 |
|
rail network 0.9951391220092773 |
|
common transport policy 0.9791265726089478 |
|
transport market 0.9368429780006409 |
|
trans-European network 0.9098047614097595 |
|
high-speed transport 0.4887568950653076 |
|
carriage of goods 0.4874659776687622 |
|
``` |
|
|
|
In German 🇩🇪: |
|
|
|
``` |
|
text = "Am 14. September 2022 schlug die Kommission eine Verordnung zum Verbot von Produkten, die unter Einsatz von Zwangsarbeit, einschließlich Kinderarbeit, hergestellt wurden, auf dem Binnenmarkt der Europäischen Union (EU) vor. Der Vorschlag bezieht sich auf alle Produkte, die auf dem EU-Markt angeboten werden, unabhängig davon, ob sie in der EU für den Inlandsverbrauch oder für die Ausfuhr hergestellt oder eingeführt werden. Er gilt für Produkte aller Art, einschließlich ihrer Bestandteile, aus allen Sektoren und Branchen. Die EU-Mitgliedstaaten wären für die Durchsetzung der Bestimmungen zuständig, und ihre nationalen Behörden könnten Produkte, die unter Einsatz von Zwangsarbeit hergestellt wurden, vom EU-Markt nehmen. Die Zollbehörden würden solche Produkte an den EU-Grenzen identifizieren und aufhalten. " |
|
|
|
goods and services 0.9618138670921326 |
|
single market 0.9268659949302673 |
|
market approval 0.6425430774688721 |
|
export restriction 0.5231644511222839 |
|
EU Member State 0.4724983870983124 |
|
free movement of goods 0.38777536153793335 |
|
electronic commerce 0.31897953152656555 |
|
``` |
|
|
|
In Bulgarian 🇧🇬: |
|
|
|
``` |
|
text = "В тази кратка бележка се обобщава проучването, в което се оценяват предизвикателствата, възможностите и средносрочните перспективи пред млечния сектор в ЕС в светлината на премахването на квотите за мляко. Проучването се фокусира върху структурните промени в сектора, динамиката на пазара на млечни продукти, необходимостта от екологична устойчивост и устойчивостта на селските райони. Разгледани са и специфичните проблеми на млечните региони в неравностойно положение. Докладът предлага политически препоръки за разглеждане от Европейския парламент с цел ефективно подпомагане на млечното животновъдство и поддържане на селските общности, като същевременно се отговори на изискванията за устойчивост на сектора." |
|
|
|
reform of the CAP 0.38253700733184814 |
|
milk 0.35211247205734253 |
|
milk product 0.2761436402797699 |
|
agricultural quota 0.24940797686576843 |
|
dairy production 0.2132476419210434 |
|
EU Member State 0.09408465027809143 |
|
``` |
|
|
|
|
|
## Architecture |
|
|
|
 |
|
|
|
This classification model is built on top of [EUBERT](https://huggingface.co/EuropeanParliament/EUBERT) with 7331 Eurovoc labels |
|
|
|
With less than 100 million parameters, it can be deployed on commodity hardware without GPU acceleration (around 200 ms per inference for 2000 characters). |
|
|
|
## Usage |
|
|
|
|
|
```python |
|
from eurovoc import EurovocTagger |
|
model = EurovocTagger.from_pretrained("EuropeanParliament/eurovoc_eu") |
|
``` |
|
see the source code also |
|
## Metrics |
|
|
|
|
|
On Eurovoc Dataset version 23.08 with a stratification ratio 90/10 for training/test and training/validation |
|
|
|
|
|
| Metric | Value | Threshold Value | |
|
|------------|------------|-----------------| |
|
| Micro F1 | 0.8188 | 0.41 | |
|
| NDCG@3 | 0.8778 | - | |
|
| NDCG@5 | 0.8625 | - | |
|
| NDCG@10 | 0.8711 | - | |
|
|
|
These values are higher than the state of the art previously known in the field, see publications: |
|
|
|
- Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2019. [Extreme Multi-Label Legal Text Classification](https://arxiv.org/abs/1905.10892): A Case Study in EU Legislation. In Proceedings of the Natural Legal Language Processing Workshop 2019, pages 78–87, Minneapolis, Minnesota. Association for Computational Linguistics. |
|
- I. Chalkidis, M. Fergadiotis, P. Malakasiotis and I. Androutsopoulos, "[Large-Scale Multi-Label Text Classification on EU Legislation](https://arxiv.org/abs/1906.02192)". Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), Florence, Italy, (short papers), 2019 () |
|
- Andrei-Marius Avram, Vasile Pais, and Dan Ioan Tufis. 2021. [PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors](https://arxiv.org/abs/2108.01139). In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 92–101, Held Online. INCOMA Ltd.. |
|
- SHAHEEN, Zein, WOHLGENANNT, Gerhard, et FILTZ, Erwin. [Large Scale Legal Text Classification Using Transformer Models](https://arxiv.org/pdf/2010.12871.pdf) |
|
|
|
|
|
These results make this model the de facto new reference in the domain. |
|
As the model is open, we encourage you to carry out your own evaluations and share them on the [discussion forum](https://huggingface.co/EuropeanParliament/eurovoc_eu/discussions) |
|
|
|
|
|
## Inference Endpoint |
|
|
|
|
|
### Payload example |
|
|
|
```json |
|
{ |
|
"inputs": "The Union condemns the continuing grave human rights violations by the Myanmar armed forces, including torture, sexual and gender-based violence, the persecution of civil society actors, human rights defenders and journalists, and attacks on the civilian population, including ethnic and religious minorities. ", |
|
"topk": 10, |
|
"threshold": 0.16 |
|
} |
|
|
|
``` |
|
|
|
result: |
|
|
|
```json |
|
{'results': [{'label': 'international sanctions', 'score': 0.9994925260543823}, |
|
{'label': 'economic sanctions', 'score': 0.9991770386695862}, |
|
{'label': 'natural person', 'score': 0.9591936469078064}, |
|
{'label': 'EU restrictive measure', 'score': 0.8388392329216003}, |
|
{'label': 'legal person', 'score': 0.45630475878715515}, |
|
{'label': 'Burma/Myanmar', 'score': 0.43375277519226074}]} |
|
``` |
|
|
|
Only six results, because the following one score is less that 0.16 |
|
|
|
Default value, topk = 5 and threshold = 0.16 |
|
|
|
|
|
## Author(s) |
|
|
|
Sébastien Campion <sebastien.campion@europarl.europa.eu> |
