|
--- |
|
language: |
|
- en |
|
- ha |
|
- yo |
|
- ig |
|
- pcm |
|
pipeline_tag: text-classification |
|
datasets: |
|
- manueltonneau/NaijaHate |
|
--- |
|
|
|
|
|
# NaijaXLM-T-base Hate |
|
This is a [NaijaXLM-T](https://huggingface.co/manueltonneau/naija-xlm-twitter-base) model finetuned on Nigerian tweets annotated for hate speech detection. The model is described and evaluated in the [reference paper](https://arxiv.org/abs/2403.19260) and was developed together with [@pvcastro](https://huggingface.co/pvcastro). |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **Model type:** xlm-roberta |
|
- **Language(s) (NLP):** (Nigerian) English, Nigerian Pidgin, Hausa, Yoruba, Igbo |
|
- **Finetuned from model:** `manueltonneau/naija-xlm-twitter-base` |
|
|
|
### Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** https://github.com/manueltonneau/hate_speech_nigeria |
|
- **Paper:** https://arxiv.org/abs/2403.19260 |
|
|
|
|
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
This model was finetuned on the stratified (`dataset=='stratified'`) and active learning (`dataset=='al'`) subset of [NaijaHate](https://huggingface.co/datasets/manueltonneau/NaijaHate). |
|
|
|
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
|
|
### Training Procedure and Evaluation |
|
|
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
We perform a 90-10 train-test split and conduct a 5-fold cross-validation with 5 learning rates ranging from 1e-5 to 5e-5. Each fold is trained using 3 different seeds. The train-test split is repeated for 10 different seeds, and the evaluation metrics are averaged across the 10 seeds. |
|
|
|
We evaluate model performance on three datasets: the holdout sample from the train-test splits as well as the top-scored sample (`dataset=='eval'`) and the random sample (`dataset=='random'`) from [NaijaHate](https://huggingface.co/datasets/manueltonneau/NaijaHate). |
|
|
|
| Model | Holdout | Top-scored | Random | |
|
|---------------|--------------------|--------------------|-------------------| |
|
| GPT-3.5, ZSL | - | 60.3±2.7 | 3.1±1.2 | |
|
| Perspective API | - | 60.2±3.5 | 4.3±2.6 | |
|
| XLM-T | *84.2 ± 0.6* | 51.8 ± 0.7 | 0.6 ± 0.1 | |
|
| XLM-T | *62.0 ± 2.3* | 68.9 ± 0.8 | 3.3 ± 0.6 | |
|
| XLM-T | *70.5 ± 3.7* | 63.7 ± 1.1 | 1.9 ± 0.5 | |
|
| DeBERTaV3 | **82.3 ± 2.3** | 85.3 ± 0.8 | **29.7 ± 4.1** | |
|
| XLM-R | 76.7 ± 2.5 | 83.6 ± 0.8 | 22.1 ± 3.7 | |
|
| mDeBERTaV3 | 29.2 ± 2.0 | 49.6 ± 1.0 | 0.2 ± 0.0 | |
|
| Conv. BERT | 79.2 ± 2.3 | 86.2 ± 0.8 | 22.6 ± 3.6 | |
|
| BERTweet | **83.6 ± 2.0** | **88.5 ± 0.6** | **34.0 ± 4.4** | |
|
| XLM-T | 79.0 ± 2.4 | 84.5 ± 0.9 | 22.5 ± 3.7 | |
|
| AfriBERTa | 70.1 ± 2.7 | 80.1 ± 0.9 | 12.5 ± 2.8 | |
|
| AfroXLM-R | 79.7 ± 2.3 | 86.1 ± 0.8 | 24.7 ± 4.0 | |
|
| XLM-R Naija | 77.0 ± 2.5 | 83.5 ± 0.8 | 19.1 ± 3.4 | |
|
| NaijaXLM-T | **83.4 ± 2.1** | **89.3 ± 0.7** | **33.7 ± 4.5** | |
|
|
|
For more information on the evaluation, please read the [reference paper](https://arxiv.org/abs/2403.19260). |
|
## BibTeX entry and citation information |
|
|
|
|
|
Please cite the [reference paper](https://arxiv.org/abs/2403.19260) if you use this model. |
|
|
|
```bibtex |
|
@article{tonneau2024naijahate, |
|
title={NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data}, |
|
author={Tonneau, Manuel and de Castro, Pedro Vitor Quinta and Lasri, Karim and Farouq, Ibrahim and Subramanian, Lakshminarayanan and Orozco-Olvera, Victor and Fraiberger, Samuel}, |
|
journal={arXiv preprint arXiv:2403.19260}, |
|
year={2024} |
|
} |
|
``` |
