Add model files for readability-es-benchmark-bertin-es-sentences-2class

README.md

@@ -1,3 +1,73 @@
 ---
 license: cc-by-4.0
 ---
+
+## Readability benchmark (ES): bertin-es-sentences-2class
+
+This project is part of a series of models from the paper "A Benchmark for Neural Readability Assessment of Texts in Spanish". 
+You can find more details about the project in our [GitHub](https://github.com/lmvasque/readability-es-benchmark).
+
+## Models
+
+Our models were fine-tuned in multiple settings, including readability assessment in 2-class (simple/complex) and 3-class (basic/intermediate/advanced) for sentences and paragraph datasets.
+You can find more details in our [paper](https://drive.google.com/file/d/1KdwvqrjX8MWYRDGBKeHmiR1NCzDcVizo/view?usp=share_link). 
+These are the available models you can use (current model page in bold):
+
+| Model                                                                                                 | Granularity    | # classes |
+|-------------------------------------------------------------------------------------------------------|----------------|:---------:|
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-2class)   | paragraphs     |     2     |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-paragraphs-3class)   | paragraphs     |     3     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class) | paragraphs     |     2     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class) | paragraphs     |     3     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class) | paragraphs |   2   |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)     | paragraphs     |     3     |
+| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-3class)  | paragraphs     |     3     |
+| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-paragraphs-2class)  | paragraphs     |     2     |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-sentences-2class)   | sentences      |     2     |
+| [BERTIN (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-sentences-3class)   | sentences      |     3     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-sentences-2class) | sentences      |     2     |
+| [BERTIN (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-bertin-es-sentences-3class) | sentences      |     3     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-sentences-2class)     | sentences      |     2     |
+| [mBERT (ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-sentences-3class)     | sentences      |     3     |
+| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-sentences-3class)  | sentences      |     3     |
+| [mBERT (EN+ES)](https://huggingface.co/lmvasque/readability-es-benchmark-mbert-es-sentences-2class)  | sentences      |     2     |
+
+For the zero-shot setting, we used the original models [BERTIN](bertin-project/bertin-roberta-base-spanish) and [mBERT](https://huggingface.co/bert-base-multilingual-uncased) with no further training.
+## Results
+
+These are our results for all the readability models in different settings. Please select your model based on the desired performance:
+
+| Granularity | Model         | F1 Score (2-class)  |  Precision (2-class)  |  Recall (2-class)  |  F1 Score (3-class)  |  Precision (3-class)  |  Recall (3-class)  |
+|-------------|---------------|:-------------------:|:---------------------:|:------------------:|:--------------------:|:---------------------:|:------------------:|
+| Paragraph   | Baseline (TF-IDF+LR) |        0.829        |         0.832         |       0.827        |        0.556         |         0.563         |       0.550        |
+| Paragraph   | BERTIN (Zero) |        0.308        |         0.222         |       0.500        |        0.227         |         0.284         |       0.338        |
+| Paragraph   | BERTIN (ES)   |        0.924        |         0.923         |       0.925        |        0.772         |         0.776         |       0.768        |
+| Paragraph   | mBERT (Zero)  |        0.308        |         0.222         |       0.500        |        0.253         |         0.312         |       0.368        |
+| Paragraph   | mBERT (EN)    |          -          |           -           |         -          |        0.505         |         0.560         |       0.552        |
+| Paragraph   | mBERT (ES)    |      **0.933**      |       **0.932**       |     **0.936**      |        0.776         |         0.777         |       0.778        |
+| Paragraph   | mBERT (EN+ES) |          -          |           -           |         -          |      **0.779**       |       **0.783**       |     **0.779**      |
+| Sentence | Baseline (TF-IDF+LR) |        0.811        |         0.814         |       0.808        |        0.525         |         0.531         |       0.521        |
+| Sentence | BERTIN (Zero) |        0.367        |         0.290         |       0.500        |        0.188         |         0.232         |       0.335        |
+| Sentence | BERTIN (ES)   |      **0.900**      |       **0.900**       |     **0.900**      |      **0.699**       |       **0.701**       |     **0.698**      |
+| Sentence | mBERT (Zero)  |        0.367        |         0.290         |       0.500        |        0.278         |         0.329         |       0.351        |
+| Sentence | mBERT (EN)    |          -          |           -           |         -          |        0.521         |         0.565         |       0.539        |
+| Sentence | mBERT (ES)    |        0.893        |         0.891         |       0.896        |        0.688         |         0.686         |       0.691        |
+| Sentence | mBERT (EN+ES) |          -          |           -           |         -          |        0.679         |         0.676         |       0.682        |
+
+
+## Citation
+
+If you use our results and scripts in your research, please cite our work: "[A Benchmark for Neural Readability Assessment of Texts in Spanish](https://drive.google.com/file/d/1KdwvqrjX8MWYRDGBKeHmiR1NCzDcVizo/view?usp=share_link)" (to be published) 
+
+```
+@inproceedings{vasquez-rodriguez-etal-2022-benchmarking,
+    title = "The Role of Text Simplification Operations in Evaluation",
+    author = "V{\'a}squez-Rodr{\'\i}guez, Laura  and
+      Cuenca-Jim{\'\e}nez, Pedro-Manuel and
+      Morales-Esquivel, Sergio Esteban and
+      Alva-Manchego, Fernando",
+    booktitle = "Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022), EMNLP 2022",
+    month = dec,
+    year = "2022",
+}
+```

config.json

@@ -0,0 +1,29 @@
+{
+  "_name_or_path": "bertin-project/bertin-roberta-base-spanish",
+  "architectures": [
+    "RobertaForSequenceClassification"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.0,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "problem_type": "single_label_classification",
+  "torch_dtype": "float32",
+  "transformers_version": "4.19.2",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 50262
+}

optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:553b1d2205c0487b78530bf87fb7b400ce0987a762747e715a074f9ccef2c631
+size 997275421

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c1f6772db53b516a6c91cb39bbf28967a28adca69c5ae2b083df6f9b991f757b
+size 498651117

rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:528887aeaf571c1dd9d1789c0fad11e336830c7f10d9174d25b3f236cf9a2aa4
+size 14503

scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dfe4967de839d5b80defe550557e070307581482fd91970422d4653d8c5f6d9e
+size 623

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true}}

tokenizer_config.json

@@ -0,0 +1 @@
+{"errors": "replace", "bos_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "sep_token": {"content": "</s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "cls_token": {"content": "<s>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "unk_token": {"content": "<unk>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "pad_token": {"content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "mask_token": {"content": "<mask>", "single_word": false, "lstrip": true, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "trim_offsets": true, "max_len": 512, "special_tokens_map_file": null, "name_or_path": "bertin-project/bertin-roberta-base-spanish", "tokenizer_class": "RobertaTokenizer"}

trainer_state.json

@@ -0,0 +1,196 @@
+{
+  "best_metric": 0.266584187746048,
+  "best_model_checkpoint": "./model/sent_2class_bertin_project_bertin_roberta_base_spanish/checkpoint-1078",
+  "epoch": 10.0,
+  "global_step": 10780,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.0,
+      "learning_rate": 2.7e-06,
+      "loss": 0.3488,
+      "step": 1078
+    },
+    {
+      "epoch": 1.0,
+      "eval_accuracy": 0.8970315398886828,
+      "eval_f1": 0.8961910338994079,
+      "eval_loss": 0.266584187746048,
+      "eval_precision": 0.8957628528645112,
+      "eval_recall": 0.8966965120197903,
+      "eval_runtime": 21.586,
+      "eval_samples_per_second": 99.88,
+      "eval_steps_per_second": 6.254,
+      "step": 1078
+    },
+    {
+      "epoch": 2.0,
+      "learning_rate": 2.4000000000000003e-06,
+      "loss": 0.2263,
+      "step": 2156
+    },
+    {
+      "epoch": 2.0,
+      "eval_accuracy": 0.898886827458256,
+      "eval_f1": 0.8977834612105712,
+      "eval_loss": 0.27391624450683594,
+      "eval_precision": 0.8984546268833581,
+      "eval_recall": 0.8972142020797937,
+      "eval_runtime": 21.5769,
+      "eval_samples_per_second": 99.922,
+      "eval_steps_per_second": 6.257,
+      "step": 2156
+    },
+    {
+      "epoch": 3.0,
+      "learning_rate": 2.1e-06,
+      "loss": 0.1778,
+      "step": 3234
+    },
+    {
+      "epoch": 3.0,
+      "eval_accuracy": 0.9025974025974026,
+      "eval_f1": 0.9005949897907656,
+      "eval_loss": 0.2852832078933716,
+      "eval_precision": 0.9078802796803653,
+      "eval_recall": 0.8972558952389886,
+      "eval_runtime": 21.6014,
+      "eval_samples_per_second": 99.808,
+      "eval_steps_per_second": 6.25,
+      "step": 3234
+    },
+    {
+      "epoch": 4.0,
+      "learning_rate": 1.8e-06,
+      "loss": 0.1403,
+      "step": 4312
+    },
+    {
+      "epoch": 4.0,
+      "eval_accuracy": 0.8882189239332097,
+      "eval_f1": 0.8875278557938429,
+      "eval_loss": 0.4905070662498474,
+      "eval_precision": 0.8867163445635435,
+      "eval_recall": 0.8889302925122561,
+      "eval_runtime": 21.5859,
+      "eval_samples_per_second": 99.88,
+      "eval_steps_per_second": 6.254,
+      "step": 4312
+    },
+    {
+      "epoch": 5.0,
+      "learning_rate": 1.5e-06,
+      "loss": 0.1057,
+      "step": 5390
+    },
+    {
+      "epoch": 5.0,
+      "eval_accuracy": 0.8979591836734694,
+      "eval_f1": 0.896845694799659,
+      "eval_loss": 0.5339986085891724,
+      "eval_precision": 0.8975153439448489,
+      "eval_recall": 0.8962778432128748,
+      "eval_runtime": 21.6294,
+      "eval_samples_per_second": 99.679,
+      "eval_steps_per_second": 6.242,
+      "step": 5390
+    },
+    {
+      "epoch": 6.0,
+      "learning_rate": 1.2000000000000002e-06,
+      "loss": 0.0784,
+      "step": 6468
+    },
+    {
+      "epoch": 6.0,
+      "eval_accuracy": 0.8979591836734694,
+      "eval_f1": 0.8963577501817979,
+      "eval_loss": 0.5496116280555725,
+      "eval_precision": 0.8998756153165819,
+      "eval_recall": 0.894290469291251,
+      "eval_runtime": 21.5964,
+      "eval_samples_per_second": 99.831,
+      "eval_steps_per_second": 6.251,
+      "step": 6468
+    },
+    {
+      "epoch": 7.0,
+      "learning_rate": 9e-07,
+      "loss": 0.0613,
+      "step": 7546
+    },
+    {
+      "epoch": 7.0,
+      "eval_accuracy": 0.8984230055658627,
+      "eval_f1": 0.8971592486956067,
+      "eval_loss": 0.6363572478294373,
+      "eval_precision": 0.8986339225376554,
+      "eval_recall": 0.8960685088094171,
+      "eval_runtime": 21.6326,
+      "eval_samples_per_second": 99.665,
+      "eval_steps_per_second": 6.241,
+      "step": 7546
+    },
+    {
+      "epoch": 8.0,
+      "learning_rate": 6.000000000000001e-07,
+      "loss": 0.0461,
+      "step": 8624
+    },
+    {
+      "epoch": 8.0,
+      "eval_accuracy": 0.900278293135436,
+      "eval_f1": 0.8989520476739503,
+      "eval_loss": 0.6520560383796692,
+      "eval_precision": 0.9009346244640362,
+      "eval_recall": 0.8975798858302324,
+      "eval_runtime": 21.6557,
+      "eval_samples_per_second": 99.558,
+      "eval_steps_per_second": 6.234,
+      "step": 8624
+    },
+    {
+      "epoch": 9.0,
+      "learning_rate": 3.0000000000000004e-07,
+      "loss": 0.0374,
+      "step": 9702
+    },
+    {
+      "epoch": 9.0,
+      "eval_accuracy": 0.897495361781076,
+      "eval_f1": 0.8959944382052516,
+      "eval_loss": 0.6639323830604553,
+      "eval_precision": 0.8988044556565169,
+      "eval_recall": 0.8942287981599419,
+      "eval_runtime": 21.6348,
+      "eval_samples_per_second": 99.654,
+      "eval_steps_per_second": 6.24,
+      "step": 9702
+    },
+    {
+      "epoch": 10.0,
+      "learning_rate": 0.0,
+      "loss": 0.0306,
+      "step": 10780
+    },
+    {
+      "epoch": 10.0,
+      "eval_accuracy": 0.8965677179962894,
+      "eval_f1": 0.8953454329048005,
+      "eval_loss": 0.6889322996139526,
+      "eval_precision": 0.8964728057179624,
+      "eval_recall": 0.8944667966103461,
+      "eval_runtime": 21.547,
+      "eval_samples_per_second": 100.06,
+      "eval_steps_per_second": 6.265,
+      "step": 10780
+    }
+  ],
+  "max_steps": 10780,
+  "num_train_epochs": 10,
+  "total_flos": 4.53813948284928e+16,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:227285920adc45962e40c3b55f40a00ae0400752e6f1232660b47e4fe96e8bc7
+size 3311