---
language:
- en
- ha
- yo
- ig
- pcm
pipeline_tag: text-classification
datasets:
- manueltonneau/NaijaHate
---


# NaijaXLM-T-base Hate
This is a [NaijaXLM-T](https://huggingface.co/manueltonneau/naija-xlm-twitter-base) model finetuned on Nigerian tweets annotated for hate speech detection. The model is described and evaluated in the [reference paper](https://arxiv.org/abs/2403.19260) and was developed together with [@pvcastro](https://huggingface.co/pvcastro). 

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

- **Model type:** xlm-roberta
- **Language(s) (NLP):** (Nigerian) English, Nigerian Pidgin, Hausa, Yoruba, Igbo
- **Finetuned from model:** `manueltonneau/naija-xlm-twitter-base`

### Model Sources [optional]

<!-- Provide the basic links for the model. -->

- **Repository:** https://github.com/manueltonneau/hate_speech_nigeria
- **Paper:** https://arxiv.org/abs/2403.19260



## Training Details

### Training Data

This model was finetuned on the stratified (`dataset=='stratified'`) and active learning (`dataset=='al'`) subset of [NaijaHate](https://huggingface.co/datasets/manueltonneau/NaijaHate).

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

### Training Procedure and Evaluation

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
We perform a 90-10 train-test split and conduct a 5-fold cross-validation with 5 learning rates ranging from 1e-5 to 5e-5. Each fold is trained using 3 different seeds. The train-test split is repeated for 10 different seeds, and the evaluation metrics are averaged across the 10 seeds.

We evaluate model performance on three datasets: the holdout sample from the train-test splits as well as the top-scored sample (`dataset=='eval'`) and the random sample (`dataset=='random'`) from [NaijaHate](https://huggingface.co/datasets/manueltonneau/NaijaHate).

 | Model         | Holdout            | Top-scored         | Random            |
|---------------|--------------------|--------------------|-------------------|
| GPT-3.5, ZSL  | -                  | 60.3±2.7           | 3.1±1.2           |
| Perspective API | -                 | 60.2±3.5           | 4.3±2.6           |
| XLM-T         | *84.2 ± 0.6*       | 51.8 ± 0.7         | 0.6 ± 0.1         |
| XLM-T         | *62.0 ± 2.3*       | 68.9 ± 0.8         | 3.3 ± 0.6         |
| XLM-T         | *70.5 ± 3.7*       | 63.7 ± 1.1         | 1.9 ± 0.5         |
| DeBERTaV3     | **82.3 ± 2.3**     | 85.3 ± 0.8         | **29.7 ± 4.1**    |
| XLM-R         | 76.7 ± 2.5         | 83.6 ± 0.8         | 22.1 ± 3.7        |
| mDeBERTaV3    | 29.2 ± 2.0         | 49.6 ± 1.0         | 0.2 ± 0.0         |
| Conv. BERT    | 79.2 ± 2.3         | 86.2 ± 0.8         | 22.6 ± 3.6        |
| BERTweet      | **83.6 ± 2.0**     | **88.5 ± 0.6**     | **34.0 ± 4.4**    |
| XLM-T         | 79.0 ± 2.4         | 84.5 ± 0.9         | 22.5 ± 3.7        |
| AfriBERTa     | 70.1 ± 2.7         | 80.1 ± 0.9         | 12.5 ± 2.8        |
| AfroXLM-R     | 79.7 ± 2.3         | 86.1 ± 0.8         | 24.7 ± 4.0        |
| XLM-R Naija   | 77.0 ± 2.5         | 83.5 ± 0.8         | 19.1 ± 3.4        |
| NaijaXLM-T | **83.4 ± 2.1**   | **89.3 ± 0.7**     | **33.7 ± 4.5**    |

For more information on the evaluation, please read the [reference paper](https://arxiv.org/abs/2403.19260). 
## BibTeX entry and citation information


Please cite the [reference paper](https://arxiv.org/abs/2403.19260) if you use this model.

```bibtex
@article{tonneau2024naijahate,
  title={NaijaHate: Evaluating Hate Speech Detection on Nigerian Twitter Using Representative Data},
  author={Tonneau, Manuel and de Castro, Pedro Vitor Quinta and Lasri, Karim and Farouq, Ibrahim and Subramanian, Lakshminarayanan and Orozco-Olvera, Victor and Fraiberger, Samuel},
  journal={arXiv preprint arXiv:2403.19260},
  year={2024}
}
```