|
--- |
|
license: cc-by-4.0 |
|
--- |
|
|
|
## Readability benchmark (ES): bertin-es-sentences-2class |
|
|
|
This project is part of a series of models from the paper "A Benchmark for Neural Readability Assessment of Texts in Spanish". |
|
You can find more details about the project in our [GitHub](https://github.com/lmvasque/readability-es-benchmark). |
|
|
|
## Models |
|
|
|
Our models were fine-tuned in multiple settings, including readability assessment in 2-class (simple/complex) and 3-class (basic/intermediate/advanced) for sentences and paragraph datasets. |
|
You can find more details in our [paper](https://drive.google.com/file/d/1KdwvqrjX8MWYRDGBKeHmiR1NCzDcVizo/view?usp=share_link). |
|
These are the available models you can use (current model page in bold): |
|
|
|
| Model | Granularity | # classes | |
|
|-----------------------------------------------------------------------------------------------------------|----------------|:---------:| |
|
| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-2class) | paragraphs | 2 | |
|
| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-3class) | paragraphs | 3 | |
|
| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class) | paragraphs | 2 | |
|
| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class) | paragraphs | 3 | |
|
| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-en-es-paragraphs-3class) | paragraphs | 3 | |
|
| **[BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-sentences-2class)** | **sentences** | **2** | |
|
| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-sentences-3class) | sentences | 3 | |
|
| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-sentences-2class) | sentences | 2 | |
|
| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-sentences-3class) | sentences | 3 | |
|
| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-en-es-sentences-3class) | sentences | 3 | |
|
|
|
|
|
For the zero-shot setting, we used the original models [BERTIN](bertin-project/bertin-roberta-base-spanish) and [mBERT](https://huggingface.co/bert-base-multilingual-uncased) with no further training. |
|
## Results |
|
|
|
These are our results for all the readability models in different settings. Please select your model based on the desired performance: |
|
|
|
| Granularity | Model | F1 Score (2-class) | Precision (2-class) | Recall (2-class) | F1 Score (3-class) | Precision (3-class) | Recall (3-class) | |
|
|-------------|---------------|:-------------------:|:---------------------:|:------------------:|:--------------------:|:---------------------:|:------------------:| |
|
| Paragraph | Baseline (TF-IDF+LR) | 0.829 | 0.832 | 0.827 | 0.556 | 0.563 | 0.550 | |
|
| Paragraph | BERTIN (Zero) | 0.308 | 0.222 | 0.500 | 0.227 | 0.284 | 0.338 | |
|
| Paragraph | BERTIN (ES) | 0.924 | 0.923 | 0.925 | 0.772 | 0.776 | 0.768 | |
|
| Paragraph | mBERT (Zero) | 0.308 | 0.222 | 0.500 | 0.253 | 0.312 | 0.368 | |
|
| Paragraph | mBERT (EN) | - | - | - | 0.505 | 0.560 | 0.552 | |
|
| Paragraph | mBERT (ES) | **0.933** | **0.932** | **0.936** | 0.776 | 0.777 | 0.778 | |
|
| Paragraph | mBERT (EN+ES) | - | - | - | **0.779** | **0.783** | **0.779** | |
|
| Sentence | Baseline (TF-IDF+LR) | 0.811 | 0.814 | 0.808 | 0.525 | 0.531 | 0.521 | |
|
| Sentence | BERTIN (Zero) | 0.367 | 0.290 | 0.500 | 0.188 | 0.232 | 0.335 | |
|
| Sentence | BERTIN (ES) | **0.900** | **0.900** | **0.900** | **0.699** | **0.701** | **0.698** | |
|
| Sentence | mBERT (Zero) | 0.367 | 0.290 | 0.500 | 0.278 | 0.329 | 0.351 | |
|
| Sentence | mBERT (EN) | - | - | - | 0.521 | 0.565 | 0.539 | |
|
| Sentence | mBERT (ES) | 0.893 | 0.891 | 0.896 | 0.688 | 0.686 | 0.691 | |
|
| Sentence | mBERT (EN+ES) | - | - | - | 0.679 | 0.676 | 0.682 | |
|
|
|
|
|
## Citation |
|
|
|
If you use our results and scripts in your research, please cite our work: "[A Benchmark for Neural Readability Assessment of Texts in Spanish](https://drive.google.com/file/d/1KdwvqrjX8MWYRDGBKeHmiR1NCzDcVizo/view?usp=share_link)" (to be published) |
|
|
|
``` |
|
@inproceedings{vasquez-rodriguez-etal-2022-benchmarking, |
|
title = "A Benchmark for Neural Readability Assessment of Texts in Spanish", |
|
author = "V{\'a}squez-Rodr{\'\i}guez, Laura and |
|
Cuenca-Jim{\'\e}nez, Pedro-Manuel and |
|
Morales-Esquivel, Sergio Esteban and |
|
Alva-Manchego, Fernando", |
|
booktitle = "Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), EMNLP 2022", |
|
month = dec, |
|
year = "2022", |
|
} |
|
``` |
|
|
